Troubleshooting in Kubernetes

Let’s go over various methods for debugging your Kubernetes application. There are numerous techniques to identify what led to the current issue. I’ll demonstrate some of the approaches I use Kubernetes in my regular work.

https://unsplash.com/photos/ifIkolRvAUg

I assume you have a containerized sample application ready to run on Kubernetes. I will deploy a sample Nginx application and make it available via Traefik reverse proxy. Here are some issues you may also encounter:

404 error, the endpoint was not found

$ curl http://myapp.example.com/version
404 page not found

The root cause of the error above could vary but let’s move step by step to narrow down the possible cause.

1.1. Validate your pod that runs your application and whether is working as expected. Forward your pod to localhost and make an API call.

$ k port-forward sample-app-5c8746dcf5-m8zg8 8080:80
$ curl localhost:8080/version

if the response is the result you expected then most probably the root cause(RC) not your application internally.

1.2. Check the defined service whether matches your pod by selector labels.

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp #this label has to match your pod selector
  ports:
    - port: 8080 #service is available from this port
      targetPort: 80 #the port your container exposed

For whatever reason if the service selector does not match the selector in the pod definition, the endpoint will be also empty as follow:

Wrong:

$ kubectl get ep
NAME         ENDPOINTS                         AGE
sample-app   <none>                            28m

Right one:

$ kubectl get ep
NAME         ENDPOINTS                         AGE
sample-app   10.244.0.18:8080                    28m

Validate also the following points:

portsName? if exist and doesnot match -> Internal Server Error?
targetPort? does not match ->  502 Bad Gateway
labels?  does not match -> 404 not found
containerPort? might get -> 502 Bad Gateway or not

To validate the service and pod are matching and functional as it should, perform as follow:

forward the service to the localhost:

$ kubectl port-forward svc/sample-app 8080

Then go to your favorite browser and hit the http://localhost:8080

If your Nginx or whatever your app is should be accessible unless the service is not configured correctly.

1. 2. Validate the Service Name that matches the service name in the Ingress declaration.

spec:
  rules:
    - host: myapp.example.com
      http:
        paths:
          - backend:
            service:
              name: myapp #hast to match service name above.
              port:
                number: 8080
            path: /
            pathType: ImplementationSpecific

The crucial is your Service name and port number have to match!

You also can make an API call on ingress and a fully qualified domain name(FQDN) as follow:

Start a curl instance on cluster and exec in, give them a try!

alias kurl='kubectl exec -it curl  -- /bin/sh' #my favorite alias
$ curl http://myapp.default.svc.cluster.local:8080/actuator/health
{"status":"UP","groups":["liveness","readiness"]}/
or
/ $ curl  http://myapp.example.com/actuator/health
{"status":"UP","groups":["liveness","readiness"]}/ $

1. 3. The might also occur during rewriting endpoints

I.e the application is available on

/api/users

you want to for whatever reason make t API call on:

/api/private/users

with Edge Stack Mapping allows you to perform this as follows:

apiVersion: getambassador.io/v3alpha1
kind:  Mapping
metadata:
  name:  httpbin-mapping
spec:
  prefix: /api/users
  rewrite: /api/private/users
  service: http://myapp:8080

if you make an API call:

curl http://myapp.example.com/api/private/users
this will be catch by edge stack and /api/private/users will be rewrite to /api/users

if somehow one of the configs in Mapping is not configured correctly, you may also get a 404 Error.

The same goes for Traefik or any other proxy server.

1. 4. Ingress Controller CRDs API version

Let’s say you have upgraded your Kubernetes version or Ingress Controller and the CRDs are not compatible with the current Kubernetes version. Please validate the ingress controller CRDs and API compatibility.

I.e Traefik helm chart version ≥ 10.0.0 require the Kubernetes version ≥1.22.0 and above. If the K8S version is lower than the required version, this also might lead 404 Error.

This was a short and specific error that was handled in this post, I hope it helps someone!