K8s Anti-Design Pattern Series – Using health probes wrongly

Introduction:

In addition to considering resource constraints, health probe settings are important to think about when migrating a service to Kubernetes.

Unless they are enabled via configuration, Kubernetes applications do not automatically include health sensors. Health probes are as important to your Kubernetes installations as resource constraints. Consequently, whenever you distribute an application to a cluster, whether it be private, public, or hybrid, it must have both resource limits and health probes.Application readiness for traffic is determined by health checks. It is important for you, as a developer, to be aware of how Kubernetes will utilise your readiness and liveness endpoints and what that implies (especially what a timeout for each one means).

Summary:

Startup probe: Sees to it that your software is properly starting up. Only one cycle is offered.

Readiness probe: Verifies that your programme can handle the requested volume of users. Always active; it never stops. If it doesn’t work, Kubernetes will cease directing users to your application (and will try later)

Liveness probe: Does a health check on your programme to make sure everything is running smoothly. Always active; it never stops. If Kubernetes detects an error, it will treat your application as if it were stalled and attempt to restart it.

Learn how each health test impacts your system and assess the top choices for your framework.

Common Pitfalls:

• The readiness check does not take into consideration third-party services (like DBS).
• Having the probe for preparation and the probe for life share the same destination
• Using a previously established “health” endpoint that was established while the programme was deployed to a virtual machine (instead of a container)
• Failure to utilize your framework’s built-in safety features (if applicable).
• The overcomplication and unpredictability of health checkups (that cause internal denial of service attacks inside the cluster)
• Bringing about a domino effect fails when the liveness health probe checks in on external services.
• Even with virtual servers and load balancers, it’s easy to accidentally set off a chain reaction of failures, which can have severe consequences (i.e., it is not specific to Kubernetes).

Use case:

In your setup, the Login service is relied upon by three other services. In a perfect world, a service’s liveness probe would only verify that the service itself is available and able to reply to queries. However, the following can occur if you configure the liveness query to verify the dependencies:

All four services are functioning properly at the outset.

There is a problem with the authentication service (perhaps the database is overloaded) and it is taking too long to process queries.The problems with the auth service are properly detected by all three services.Unfortunately, they have each put themselves in a down (or unavailable) health state even though they are perfectly functional.When Kubernetes’ liveness sensors detect that all four services are offline, it immediately reboots them (while in reality only one of them had issues).Since circuit breakers let you separate the “liveness” of an application from its responsiveness, you should become acquainted with them for more complex cases.