A Kubernetes service defines a logical set of pods that can be accessed either from other pods in the cluster, or from outside of the cluster. Issues with services often directly impact users by causing APIs to be unavailable.


Empty Service


When a service is defined without a selector, or with a selector that does not match any pods, then there will not be any endpoints created for that service. This is an issue when that service is expected to actually send traffic to pods. Double-check the service configuration to make sure the selector is defined correctly, or manually create the endpoints for the service.

 

Unhealthy Service


The Blue Matador agent will periodically iterate through every endpoint of every service and attempt to connect over each port defined in the service. This check can expose a variety of issues:

  • Incorrect port defined in service
  • Incorrect port exposed by pod
  • Application on pod not listening on correct port
  • Security group not allowing communication between nodes
  • High network latency between nodes

If the port of an endpoint for a service is not reachable, the service is considered unhealthy. 

To debug an unhealthy service, perform the following checks:

  • Make sure the ports defined in the service match the ports exposed by the targeted pods
  • Open a shell on the pod using kubectl exec -it <pod name> -- /bin/sh and try to see if the expected ports are open on the pod. You can use the telnet, netstat, or nc tools to do this on the pod.
  • Ensure the expected applications are running on the pod and listening on the correct ports
  • Try to reach the affected pod by its ip directly from another pod, preferably on another node. You can get this information about pods by using kubectl get pods -o wide
  • If you are unable to connect between nodes, you can attempt to ssh directly to the nodes and test network connectivity from that level as well

To ensure that a service does not contain endpoints for containers that do not listen on the expected port, you can define a readiness probe in the container spec.  When the container does not match the conditions for the readiness probe, its pod will be removed as a target by the service.

 

RESOURCES