12 - Health Checks & Probes
📋 Jump to Takeaways🎁 Your container is running and responding to pings, so why is Kubernetes restarting it?
Kubernetes says the pod is Running. Users are getting 502s.
The process is alive. It answers pings. A goroutine acquired a lock and never released it, and now every incoming request reaches that same lock and waits. The process is not processing anything. You had no liveness probe.
Now picture the opposite. You added a liveness probe after that incident. The probe checks your database to confirm the app can actually do work. Three weeks later, the database has a brief network hiccup, 15 seconds of elevated latency. All 8 pods fail their liveness checks. All 8 pods restart simultaneously. They all try to reconnect to the database at once. The database, already struggling, gets hammered with 8 reconnection bursts. A 15-second blip becomes a 4-minute outage.
These are the two classic probe failures: no liveness at all, and liveness that checks shared dependencies. The solution is not "add more probes." It's understanding what each probe type is for.
This lesson adds devctl health check <service> to the tool — a command that polls a service's dependencies, reports aggregate status, and gives you a clear answer before you ship a deploy.
Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.
Why Not Just Check If the Process Is Running?
If the pod is running, shouldn't Kubernetes already know it's healthy?
A process check tells you whether the binary is still executing. It says nothing about whether it's doing anything useful. A deadlocked goroutine doesn't exit. A hung connection pool doesn't exit. A startup failure that left internal state broken doesn't exit. The process stays up, Kubernetes sees "Running," and users get errors.
Kubernetes can only know your app is healthy if you tell it what "healthy" means. That's what probes do.
Liveness vs Readiness vs Startup Probes
Three probes answer three different questions. Get them backwards and you get either undetected deadlocks or cascading restarts.
- Liveness: "Is this process stuck?" Kubernetes restarts the container on failure. Use for deadlocks, infinite loops, and corrupted internal state.
- Readiness: "Can this instance serve traffic right now?" Kubernetes removes the pod from the service endpoints on failure. Use for dependency unavailability, warmup, and backpressure.
- Startup: "Has this application finished initializing?" Checked only during startup. Once it passes, liveness and readiness take over. Use for slow-starting apps that would fail their liveness probe during a normal boot sequence.
The most common mistake is putting a database check in the liveness probe. When the database has a network hiccup, all pods fail liveness simultaneously. Kubernetes restarts all of them at once. They all reconnect to the database at the same moment. What was a 15-second blip becomes a full reconnection storm.
Readiness does the right thing here. When the database is slow or unreachable, pods are removed from the load balancer until the database recovers. They come back one by one as the readiness probe starts passing again. No restarts, no storm. The readiness probe turns a dependency outage into a graceful degradation instead of a cascading failure.
Keep liveness trivial. If your liveness probe fails, pods die. If your readiness probe fails, pods wait.
Building Health Check Endpoints
Separate endpoints let Kubernetes ask different questions without conflating the answers. The Checker struct runs any registered check function with a timeout, in parallel. Running checks in parallel matters: if your database takes 2 seconds to respond and your Redis takes 1 second, sequential checks mean a 3-second health endpoint. With goroutines, it's 2 seconds.
package health
import (
"encoding/json"
"net/http"
"sync"
"time"
)
type Status string
const (
StatusUp Status = "up"
StatusDown Status = "down"
StatusDegraded Status = "degraded"
)
type CheckResult struct {
Name string `json:"name"`
Status Status `json:"status"`
DurationMs int64 `json:"duration_ms"` // elapsed time in milliseconds
Error string `json:"error,omitempty"`
}
type HealthResponse struct {
Status Status `json:"status"`
Checks []CheckResult `json:"checks,omitempty"`
Version string `json:"version,omitempty"`
}
type Check struct {
Name string
Fn func() error
Timeout time.Duration
}
type Checker struct {
checks []Check
mu sync.RWMutex
}
func NewChecker() *Checker {
return &Checker{}
}
func (c *Checker) AddCheck(name string, fn func() error, timeout time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.checks = append(c.checks, Check{Name: name, Fn: fn, Timeout: timeout})
}
func (c *Checker) RunChecks() HealthResponse {
c.mu.RLock()
checks := make([]Check, len(c.checks))
copy(checks, c.checks)
c.mu.RUnlock()
results := make([]CheckResult, len(checks))
var wg sync.WaitGroup
for i, check := range checks {
wg.Add(1)
go func(idx int, chk Check) {
defer wg.Done()
start := time.Now()
errCh := make(chan error, 1)
go func() { errCh <- chk.Fn() }()
select {
case err := <-errCh:
result := CheckResult{
Name: chk.Name,
DurationMs: time.Since(start).Milliseconds(),
Status: StatusUp,
}
if err != nil {
result.Status = StatusDown
result.Error = err.Error()
}
results[idx] = result
case <-time.After(chk.Timeout):
results[idx] = CheckResult{
Name: chk.Name,
Status: StatusDown,
DurationMs: chk.Timeout.Milliseconds(),
Error: "timeout",
}
}
}(i, check)
}
wg.Wait()
// Aggregate overall status
overall := StatusUp
for _, r := range results {
if r.Status == StatusDown {
overall = StatusDown
break
}
}
return HealthResponse{Status: overall, Checks: results}
}Dependency Health Checks
Every external dependency needs its own check with its own timeout. If you bundle them without timeouts, a single slow dependency blocks the entire health response.
Redis speaks RESP, a simple text protocol over TCP. You could implement a basic PING with net.Conn directly. go-redis is worth it here because it handles connection pooling, reusing connections across health checks instead of opening a new TCP connection each time. For a health aggregator that fires checks every few seconds, a pooled client avoids unnecessary connection overhead.
package health
import (
"context"
"database/sql"
"fmt"
"net/http"
"time"
"github.com/redis/go-redis/v9"
)
// Database check: verify the connection pool can execute a query
func DatabaseCheck(db *sql.DB) func() error {
return func() error {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
return db.PingContext(ctx)
}
}
// Redis check: verify the cache is reachable
func RedisCheck(client *redis.Client) func() error {
return func() error {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
return client.Ping(ctx).Err()
}
}
// External API check: verify a dependency responds
func HTTPCheck(url string, expectedStatus int) func() error {
client := &http.Client{Timeout: 5 * time.Second}
return func() error {
resp, err := client.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != expectedStatus {
return fmt.Errorf("expected %d, got %d", expectedStatus, resp.StatusCode)
}
return nil
}
}Register them:
checker := health.NewChecker()
checker.AddCheck("postgres", health.DatabaseCheck(db), 3*time.Second)
checker.AddCheck("redis", health.RedisCheck(redisClient), 2*time.Second)
checker.AddCheck("payment-api", health.HTTPCheck("https://api.payments.io/health", 200), 5*time.Second)DatabaseCheck and RedisCheck each wrap their calls in context.WithTimeout. The Checker also enforces a timeout via time.After. Set the individual check timeouts tighter than the Checker timeout. The Checker timeout is the upper bound; the individual timeout is the expected bound.
Circuit Breaker Pattern
When a dependency is failing, you don't want every health check hammering it. A database timeout takes 2 seconds. If your health endpoint fires every 5 seconds and the database is down, you spend 40% of your time waiting for timeouts. A circuit breaker stops calling a failing dependency and returns an immediate error until the system recovers:
package circuitbreaker
import (
"errors"
"sync"
"time"
)
type State int
const (
Closed State = iota // Normal operation, requests pass through
Open // Failing, requests blocked
HalfOpen // Testing, one request allowed through
)
var ErrCircuitOpen = errors.New("circuit breaker is open")
type CircuitBreaker struct {
mu sync.Mutex
state State
failures int
threshold int
lastFailure time.Time
resetTimeout time.Duration
}
func New(threshold int, resetTimeout time.Duration) *CircuitBreaker {
return &CircuitBreaker{
state: Closed,
threshold: threshold,
resetTimeout: resetTimeout,
}
}
func (cb *CircuitBreaker) Execute(fn func() error) error {
cb.mu.Lock()
switch cb.state {
case Open:
// Check if reset timeout elapsed
if time.Since(cb.lastFailure) > cb.resetTimeout {
cb.state = HalfOpen
cb.mu.Unlock()
return cb.tryExec(fn)
}
cb.mu.Unlock()
return ErrCircuitOpen
case HalfOpen:
cb.mu.Unlock()
return cb.tryExec(fn)
default: // Closed
cb.mu.Unlock()
return cb.tryExec(fn)
}
}
func (cb *CircuitBreaker) tryExec(fn func() error) error {
err := fn()
cb.mu.Lock()
defer cb.mu.Unlock()
if err != nil {
cb.failures++
cb.lastFailure = time.Now()
if cb.failures >= cb.threshold {
cb.state = Open
}
return err
}
// Success: reset
cb.failures = 0
cb.state = Closed
return nil
}
func (cb *CircuitBreaker) State() State {
cb.mu.Lock()
defer cb.mu.Unlock()
return cb.state
}Wrap dependency checks with a circuit breaker:
dbBreaker := circuitbreaker.New(3, 30*time.Second) // open after 3 failures, retry after 30s
checker.AddCheck("postgres", func() error {
return dbBreaker.Execute(func() error {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
return db.PingContext(ctx)
})
}, 3*time.Second)When the database is down, the circuit opens after 3 consecutive failures. Subsequent checks return immediately with ErrCircuitOpen instead of waiting for timeouts. After 30 seconds, it allows one probe through. If that succeeds, normal operation resumes.
Graceful Degradation
Not every failed dependency means your app is down. If Redis is unavailable but your app can fall back to the database, it's degraded, not dead. Modeling that distinction in your health response lets Kubernetes make better decisions:
func (c *Checker) RunChecksWithDegradation(critical []string) HealthResponse {
resp := c.RunChecks()
criticalSet := make(map[string]bool)
for _, name := range critical {
criticalSet[name] = true
}
overall := StatusUp
for _, check := range resp.Checks {
if check.Status == StatusDown {
if criticalSet[check.Name] {
overall = StatusDown
break
}
overall = StatusDegraded
}
}
resp.Status = overall
return resp
}StatusDegraded from readiness keeps the pod in the load balancer but signals to monitoring that something is wrong. StatusDown from readiness removes the pod from traffic until the critical dependency recovers.
Kubernetes Probe Configuration
Wire the health checker into HTTP handlers and configure Kubernetes to use them. The /healthz liveness endpoint should never touch a database. The /readyz readiness endpoint should, because readiness is what protects you from serving traffic when your dependencies are down:
func main() {
checker := health.NewChecker()
// ... register checks ...
mux := http.NewServeMux()
// Liveness: is the process alive? Keep this simple.
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"status":"up"}`))
// Output: {"status":"up"}
})
// Readiness: can we serve traffic?
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
resp := checker.RunChecksWithDegradation([]string{"postgres"})
status := http.StatusOK
if resp.Status == health.StatusDown {
status = http.StatusServiceUnavailable
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
json.NewEncoder(w).Encode(resp)
// Output: {"status":"up","checks":[{"name":"postgres","status":"up","duration_ms":3},{"name":"redis","status":"up","duration_ms":1}]}
})
// Aggregate status: poll downstream services and report combined health
mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
services := []health.ServiceHealth{
{Name: "api", URL: "http://api:8080/healthz", Status: health.StatusUp},
{Name: "worker", URL: "http://worker:8080/healthz", Status: health.StatusUp},
{Name: "scheduler", URL: "http://scheduler:8080/healthz", Status: health.StatusDegraded},
}
overall := health.AggregateHealth(services)
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(map[string]any{"status": overall, "services": services})
// Output: {"services":[...],"status":"degraded"}
})
fmt.Println("Health server listening on :8080") // Output: Health server listening on :8080
if err := http.ListenAndServe(":8080", mux); err != nil {
log.Fatal(err)
}
}The Kubernetes deployment spec:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
image: myapp:latest
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 2
startupProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 0
periodSeconds: 2
failureThreshold: 30The startup probe gives the app up to 60 seconds (30 checks at 2-second intervals) to initialize before liveness kicks in. Without it, a slow-starting app gets killed by the liveness probe before it finishes booting.
Health Check Aggregation
For systems with many services, aggregate individual service health into a single endpoint operators can check without polling each service individually:
type ServiceHealth struct {
Name string `json:"name"`
URL string `json:"-"`
Status Status `json:"status"`
Error string `json:"error,omitempty"`
}
func AggregateHealth(services []ServiceHealth) Status {
downCount := 0
degradedCount := 0
for _, svc := range services {
switch svc.Status {
case StatusDown:
downCount++
case StatusDegraded:
degradedCount++
}
}
switch {
case downCount == len(services):
return StatusDown
case downCount > 0 || degradedCount > 0:
return StatusDegraded
default:
return StatusUp
}
}Expose this on a /status endpoint in a gateway or status service. Operators get one view of system health without checking each service individually.
Putting It Together: the Health Check Command
Wire the checker as the devctl health check subcommand:
func newHealthCheckCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "check <service>",
Short: "Run health checks against a service's dependencies",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
checker := buildChecker(args[0]) // returns a configured *health.Checker
resp := checker.RunChecksWithDegradation([]string{"postgres"})
return printHealthResult(resp)
},
}
return cmd
}
var healthCmd = &cobra.Command{Use: "health", Short: "Health check commands"}
func init() {
healthCmd.AddCommand(newHealthCheckCmd())
rootCmd.AddCommand(healthCmd)
}Go back to the two failures from the intro. With this system in place, both scenarios end differently.
The deadlock scenario: the goroutine is stuck, the server stops processing requests. The /healthz liveness probe makes an HTTP round-trip through the server. It times out after failureThreshold attempts. Kubernetes restarts that one pod. Not all eight. Just the one that failed.
The database hiccup scenario: the database becomes slow. The /readyz readiness probe runs DatabaseCheck, which times out after 2 seconds. The circuit breaker counts the failure. After 3 failures, it opens, and subsequent checks return immediately. Kubernetes removes affected pods from the load balancer, but does not restart them. When the database recovers, the circuit breaker lets one probe through, it succeeds, the pods re-enter the load balancer. The database never saw 8 simultaneous reconnection storms.
The only change required in the Kubernetes spec is: /healthz has no dependency checks, /readyz has all of them.
Health Aggregator Service
A combined service integrating dependency health checks (database, Redis, external APIs), a circuit breaker to prevent cascading timeouts, graceful degradation for non-critical dependencies, and a health aggregation layer for multi-service monitoring. Exposes /healthz, /readyz, and /status endpoints suitable for Kubernetes probes and operator dashboards.
Input: Dependency connection parameters (database DSN, Redis address, API URLs) and circuit breaker configuration (failure threshold, reset timeout).
Output: Serves liveness, readiness, and aggregation endpoints on :8080. Returns structured JSON with per-dependency status, overall system health (up/degraded/down), and circuit breaker state.
Full source: examples/health-aggregator
Key Takeaways
- Liveness probes answer "is it stuck?" and trigger restarts on failure. Readiness probes answer "can it serve traffic?" and remove pods from the load balancer on failure.
- Never put a database or shared dependency check in the liveness probe. A brief DB outage should remove pods from traffic, not restart all of them simultaneously.
- Run dependency checks concurrently with timeouts. A slow check should not block the health response.
- Circuit breakers prevent failing dependencies from causing cascading timeouts. Open after N failures, retry after a cooldown.
- Distinguish critical from non-critical dependencies. Redis down is degraded. Database down is unready.
- Keep
/healthztrivial. Put real dependency logic in/readyz. - Use startup probes for slow-starting apps. Without one, a slow boot gets killed by the liveness probe.
- Return structured JSON from health endpoints. It's useful for monitoring dashboards and automated tooling, not just Kubernetes.
🎁 devctl health check confirms your dependencies are ready. Next: devctl pipeline run <pipeline.yaml> — run your CI/CD pipeline locally with the same binary you push to CI.