Health Probes
Overview
Health probes provide an automated way for SaladCloud users to trigger certain actions based on the status of the containers in a container group deployment. SCE supports three types of probes: Startup, Liveness, and Readiness probes.
These probes run as “commands” in each container deployed deployed in the container group. If configured, the Startup probe runs first, and the Liveness/Readiness probes (if configured) are suspended until after the Startup probe is successful.
Caution
Health probes can be extremely useful for catching deadlocks in an application, recovering from errors during container initialization, and ensuring local resources are available before exposing nodes via networking.
However, misconfigured health probes can easily cause unnecessary termination of containers, leading to slower deployment and less reliable application uptime. It is important to first understand the normal behavior of your own containers on SaladCloud in order to minimize false positives when configuring health probes.
Probe States
Each probe can be in 1 of 3 states:
Unknown- The container is neither in a success or failure state yet. Causes no change to the system.Success- The probe has met thesuccessThresholdand has not yet failed.Failure- The container failed the diagnostic. In the case of a failed Startup or Liveness probe, the container will be automatically reallocated to a new node. In the case of a failed Readiness probe, the container will continue to run on the node, but no networking traffic will be routed to it.
Enabling Health Probes
Each of these probes can be configured from the container group configuration page by clicking the edit button beside each probe section.
Probe Configuration
Currently, each probe supports the exec protocol, which simply refers to the execution of a command within the container. We plan to add support for additional protocols (gRPC, TCP, HTTP) in the future.
To perform a probe, Salad executes the command you specify in the target container. To configure a command, press the Edit link as shown below.
Configure the command and any additional arguments. In this example, the container will attempt to read a file located at /tmp/healthy exactly Initial Delay Seconds after startup. If it successfully executes the command within Timeout Seconds, the probe returns an exit code 0. Once this has happened Success Threshold times, the Startup probe is 'done' and the Liveness and Readiness probes (if configured) are initiated.
Standard Probe Properties
All probes (Startup, Liveness, and Readiness) share the following properties.
| Property | Type (Min, Max) | Details |
|---|---|---|
| Initial Delay Seconds | Integer (0 - 10,000) | The number of seconds after the container has started before the probe is initiated. If a Startup probe is configured, the initial delay for Liveness and Readiness probes begins counting when the Startup probe has reached the success threshold. |
| Period Seconds | Integer (1 - 10,000) | How often, in seconds, to perform the probe. |
| Timeout Seconds | Integer (1 - 10,000) | After a probe initiates, the number of seconds to wait for a successful response before timing out (failing). |
| Success Threshold | Integer (1 - 10,000) | The minimum consecutive successes for the probe to return a 0 (success) for the probe to be considered successful. |
| Failure Threshold | Integer (1 - 10,000) | The number of consecutive failures of the probe before the container is reallocated (in the case of Startup and Liveness probes) or networking is disabled (in the case of Readiness probes). |
Further Reading
In SCE, the Startup, Liveness, and Readiness probes were designed based on the Kubernetes specifications. For additional information on how probes work under the hood, as well as excellent examples of configured probes and common pitfalls, check out the Kubernetes documentation on configuring probes and when to use each one.
Updated 16 days ago