12 - Health Checks & Probes

🎁 Your container is running and responding to pings, so why is Kubernetes restarting it?

A running process isn't necessarily a healthy one. Your container might be up but deadlocked. Your app might respond to requests but can't reach the database.

Kubernetes probes and health check endpoints solve this. They give orchestrators the information they need to route traffic, restart containers, and maintain system reliability.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Liveness vs Readiness vs Startup Probes

Kubernetes defines three probe types, each answering a different question:

Liveness: "Is this process stuck?" If it fails, Kubernetes restarts the container. Use for detecting deadlocks, infinite loops, or corrupted state.
Readiness: "Can this instance serve traffic?" If it fails, Kubernetes removes the pod from the Service endpoints. Use for dependency unavailability, warmup periods, or backpressure.
Startup: "Has this application finished initializing?" Only checked during startup. Once it passes, liveness and readiness take over. Use for slow-starting applications.

The critical distinction: liveness failure kills the pod. Readiness failure stops traffic. Getting these backwards causes cascading failures. If your readiness probe checks a shared database and the database goes down, all pods get restarted simultaneously instead of just being removed from load balancing.

Building Health Check Endpoints

A basic health check system with separate liveness and readiness endpoints:

package health

import (
	"encoding/json"
	"net/http"
	"sync"
	"time"
)

type Status string

const (
	StatusUp       Status = "up"
	StatusDown     Status = "down"
	StatusDegraded Status = "degraded"
)

type CheckResult struct {
	Name       string `json:"name"`
	Status     Status `json:"status"`
	DurationMs int64  `json:"duration_ms"` // elapsed time in milliseconds
	Error      string `json:"error,omitempty"`
}

type HealthResponse struct {
	Status  Status        `json:"status"`
	Checks  []CheckResult `json:"checks,omitempty"`
	Version string        `json:"version,omitempty"`
}

type Check struct {
	Name    string
	Fn      func() error
	Timeout time.Duration
}

type Checker struct {
	checks []Check
	mu     sync.RWMutex
}

func NewChecker() *Checker {
	return &Checker{}
}

func (c *Checker) AddCheck(name string, fn func() error, timeout time.Duration) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.checks = append(c.checks, Check{Name: name, Fn: fn, Timeout: timeout})
}

func (c *Checker) RunChecks() HealthResponse {
	c.mu.RLock()
	checks := make([]Check, len(c.checks))
	copy(checks, c.checks)
	c.mu.RUnlock()

	results := make([]CheckResult, len(checks))
	var wg sync.WaitGroup

	for i, check := range checks {
		wg.Add(1)
		go func(idx int, chk Check) {
			defer wg.Done()
			start := time.Now()

			errCh := make(chan error, 1)
			go func() { errCh <- chk.Fn() }()

			select {
			case err := <-errCh:
				result := CheckResult{
					Name:     chk.Name,
					DurationMs: time.Since(start).Milliseconds(),
					Status:   StatusUp,
				}
				if err != nil {
					result.Status = StatusDown
					result.Error = err.Error()
				}
				results[idx] = result
			case <-time.After(chk.Timeout):
				results[idx] = CheckResult{
					Name:     chk.Name,
					Status:   StatusDown,
					DurationMs: chk.Timeout.Milliseconds(),
					Error:    "timeout",
				}
			}
		}(i, check)
	}

	wg.Wait()

	// Aggregate overall status
	overall := StatusUp
	for _, r := range results {
		if r.Status == StatusDown {
			overall = StatusDown
			break
		}
	}

	return HealthResponse{Status: overall, Checks: results}
}

Dependency Health Checks

Real applications depend on databases, caches, and external APIs. Each gets its own check function:

package health

import (
	"context"
	"database/sql"
	"fmt"
	"net/http"
	"time"

	"github.com/redis/go-redis/v9"
)

// Database check: verify the connection pool can execute a query
func DatabaseCheck(db *sql.DB) func() error {
	return func() error {
		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		defer cancel()
		return db.PingContext(ctx)
	}
}

// Redis check: verify the cache is reachable
func RedisCheck(client *redis.Client) func() error {
	return func() error {
		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		defer cancel()
		return client.Ping(ctx).Err()
	}
}

// External API check: verify a dependency responds
func HTTPCheck(url string, expectedStatus int) func() error {
	client := &http.Client{Timeout: 5 * time.Second}
	return func() error {
		resp, err := client.Get(url)
		if err != nil {
			return err
		}
		defer resp.Body.Close()
		if resp.StatusCode != expectedStatus {
			return fmt.Errorf("expected %d, got %d", expectedStatus, resp.StatusCode)
		}
		return nil
	}
}

checker := health.NewChecker()
checker.AddCheck("postgres", health.DatabaseCheck(db), 3*time.Second)
checker.AddCheck("redis", health.RedisCheck(redisClient), 2*time.Second)
checker.AddCheck("payment-api", health.HTTPCheck("https://api.payments.io/health", 200), 5*time.Second)

Circuit Breaker Pattern

When a dependency is failing, you don't want every health check (and every request) hammering it. A circuit breaker stops calling a failing dependency and returns an immediate error until the system recovers:

package circuitbreaker

import (
	"errors"
	"sync"
	"time"
)

type State int

const (
	Closed   State = iota // Normal operation, requests pass through
	Open                  // Failing, requests blocked
	HalfOpen              // Testing, one request allowed through
)

var ErrCircuitOpen = errors.New("circuit breaker is open")

type CircuitBreaker struct {
	mu           sync.Mutex
	state        State
	failures     int
	threshold    int
	lastFailure  time.Time
	resetTimeout time.Duration
}

func New(threshold int, resetTimeout time.Duration) *CircuitBreaker {
	return &CircuitBreaker{
		state:        Closed,
		threshold:    threshold,
		resetTimeout: resetTimeout,
	}
}

func (cb *CircuitBreaker) Execute(fn func() error) error {
	cb.mu.Lock()

	switch cb.state {
	case Open:
		// Check if reset timeout elapsed
		if time.Since(cb.lastFailure) > cb.resetTimeout {
			cb.state = HalfOpen
			cb.mu.Unlock()
			return cb.tryExec(fn)
		}
		cb.mu.Unlock()
		return ErrCircuitOpen

	case HalfOpen:
		cb.mu.Unlock()
		return cb.tryExec(fn)

	default: // Closed
		cb.mu.Unlock()
		return cb.tryExec(fn)
	}
}

func (cb *CircuitBreaker) tryExec(fn func() error) error {
	err := fn()

	cb.mu.Lock()
	defer cb.mu.Unlock()

	if err != nil {
		cb.failures++
		cb.lastFailure = time.Now()
		if cb.failures >= cb.threshold {
			cb.state = Open
		}
		return err
	}

	// Success: reset
	cb.failures = 0
	cb.state = Closed
	return nil
}

func (cb *CircuitBreaker) State() State {
	cb.mu.Lock()
	defer cb.mu.Unlock()
	return cb.state
}

Wrap dependency checks with a circuit breaker:

dbBreaker := circuitbreaker.New(3, 30*time.Second) // open after 3 failures, retry after 30s

checker.AddCheck("postgres", func() error {
	return dbBreaker.Execute(func() error {
		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		defer cancel()
		return db.PingContext(ctx)
	})
}, 3*time.Second)

When the database is down, the circuit breaker opens after 3 consecutive failures. Subsequent checks return immediately with ErrCircuitOpen instead of waiting for timeouts. After 30 seconds, it allows one probe through. If that succeeds, normal operation resumes.

Graceful Degradation

Not every failed dependency means your app is unhealthy. If Redis is down but your app can fall back to the database, it's degraded, not dead. Model this in your health response:

func (c *Checker) RunChecksWithDegradation(critical []string) HealthResponse {
	resp := c.RunChecks()

	criticalSet := make(map[string]bool)
	for _, name := range critical {
		criticalSet[name] = true
	}

	overall := StatusUp
	for _, check := range resp.Checks {
		if check.Status == StatusDown {
			if criticalSet[check.Name] {
				overall = StatusDown
				break
			}
			overall = StatusDegraded
		}
	}
	resp.Status = overall
	return resp
}

Return StatusDegraded from readiness but still pass liveness. The pod stays alive but may be excluded from traffic depending on your probe configuration.

Kubernetes Probe Configuration

Wire the health checker into HTTP handlers and configure Kubernetes to call them:

func main() {
	checker := health.NewChecker()
	// ... register checks ...

	mux := http.NewServeMux()

	// Liveness: is the process alive? Keep this simple.
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte(`{"status":"up"}`))
		// Output: {"status":"up"}
	})

	// Readiness: can we serve traffic?
	mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
		resp := checker.RunChecksWithDegradation([]string{"postgres"})
		status := http.StatusOK
		if resp.Status == StatusDown {
			status = http.StatusServiceUnavailable
		}
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(status)
		json.NewEncoder(w).Encode(resp)
		// Output: {"status":"up","checks":[{"name":"postgres","status":"up","duration_ms":3},{"name":"redis","status":"up","duration_ms":1}]}
	})

	fmt.Println("Health server listening on :8080") // Output: Health server listening on :8080
	http.ListenAndServe(":8080", mux)
}

The Kubernetes deployment spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:latest
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 2
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 2
            failureThreshold: 30

The startup probe gives the app up to 60 seconds (30 × 2s) to initialize before liveness kicks in. This prevents slow-starting apps from getting killed during startup.

Health Check Aggregation

For systems with many microservices, aggregate individual service health into a single dashboard endpoint:

type ServiceHealth struct {
	Name   string `json:"name"`
	URL    string `json:"-"`
	Status Status `json:"status"`
	Error  string `json:"error,omitempty"`
}

func AggregateHealth(services []ServiceHealth) Status {
	downCount := 0
	for _, svc := range services {
		if svc.Status == StatusDown {
			downCount++
		}
	}

	switch {
	case downCount == 0:
		return StatusUp
	case downCount == len(services):
		return StatusDown
	default:
		return StatusDegraded
	}
}

Expose this on a /status endpoint in a gateway or status service. Operators get a single view of system health without checking each service individually.

Health Aggregator Service

A combined service integrating dependency health checks (database, Redis, external APIs), a circuit breaker to prevent cascading timeouts, graceful degradation for non-critical dependencies, and a health aggregation layer for multi-service monitoring. Exposes /healthz, /readyz, and /status endpoints suitable for Kubernetes probes and operator dashboards.

Input: Dependency connection parameters (database DSN, Redis address, API URLs) and circuit breaker configuration (failure threshold, reset timeout).

Output: Serves liveness, readiness, and aggregation endpoints on :8080. Returns structured JSON with per-dependency status, overall system health (up/degraded/down), and circuit breaker state.

Full source: examples/health-aggregator

Key Takeaways

Liveness probes answer "is it stuck?" (restart on failure). Readiness probes answer "can it serve?" (remove from LB on failure). Never use readiness to check shared dependencies; a database outage shouldn't restart every pod.
Run dependency checks concurrently with timeouts. A slow check shouldn't block the health response.
Circuit breakers prevent failing dependencies from causing cascading timeouts. Open the circuit after N failures, retry after a cooldown.
Distinguish critical vs non-critical dependencies. Redis down = degraded. Database down = unready.
Keep liveness probes trivial (just return 200). Put real logic in readiness probes.
Use startup probes for slow-starting applications to avoid premature restarts during initialization.
Return structured JSON from health endpoints. It's useful for monitoring dashboards and automated tooling, not just Kubernetes.

🎁 What if you could build your own CI runner that parses pipeline YAML, executes steps as isolated subprocesses, and reports status via webhooks?