14 - Deployment Automation

📋 Jump to Takeaways

🎁 What if you could replace every running instance of your app with a new version, and not a single user notices anything happened?

Deploying software reliably requires more than pushing new binaries to servers. Production systems need controlled rollout strategies that minimize blast radius, verify health at each step, and automatically revert when something goes wrong. The three dominant strategies (rolling, blue-green, and canary) each trade off speed, safety, and complexity differently.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Deployment Strategies Overview

Three dominant strategies control how traffic shifts from old to new versions:

Strategy	Mechanism	Risk	Rollback Speed
Rolling	Replace instances one-by-one	Gradual	Moderate (re-deploy old)
Blue-Green	Run two identical environments, switch atomically	Low (full pre-validation)	Instant (switch back)
Canary	Route small % of traffic to new version	Very low	Instant (route 0%)

All three share a core principle: never commit fully to a new version until health is verified.

Instance and Health Check Model

First, define the types that represent your deployment targets:

package deploy

import (
    "fmt"
    "net/http"
    "time"
)

type Instance struct {
    ID       string
    Address  string
    Version  string
    Healthy  bool
}

type HealthChecker struct {
    Client   *http.Client
    Endpoint string // e.g., "/healthz"
    Retries  int
    Interval time.Duration
}

func (h *HealthChecker) Check(instance Instance) error {
    url := fmt.Sprintf("http://%s%s", instance.Address, h.Endpoint)

    for attempt := 0; attempt <= h.Retries; attempt++ {
        resp, err := h.Client.Get(url)
        if err == nil && resp.StatusCode == http.StatusOK {
            resp.Body.Close()
            return nil
        }
        if resp != nil {
            resp.Body.Close()
        }

        if attempt < h.Retries {
            time.Sleep(h.Interval)
        }
    }
    return fmt.Errorf("instance %s unhealthy after %d attempts", instance.ID, h.Retries+1)
}

Rolling Update Implementation

Rolling updates replace instances one at a time, verifying health after each replacement before proceeding:

package deploy

import (
    "context"
    "fmt"
    "log"
)

type RollingDeployer struct {
    Health     *HealthChecker
    MaxFailed  int // max consecutive failures before abort
}

type DeployAction func(ctx context.Context, instance Instance, newVersion string) error

func (d *RollingDeployer) Deploy(
    ctx context.Context,
    instances []Instance,
    newVersion string,
    deployFn DeployAction,
) (*DeployResult, error) {
    result := &DeployResult{
        Strategy:   "rolling",
        NewVersion: newVersion,
    }

    consecutiveFailures := 0

    for i, inst := range instances {
        log.Printf("[%d/%d] Deploying %s: %s → %s",
            i+1, len(instances), inst.ID, inst.Version, newVersion)
        // Output: [1/4] Deploying web-01: v1.2.0 → v1.3.0

        // Deploy the new version to this instance
        if err := deployFn(ctx, inst, newVersion); err != nil {
            result.FailedInstances = append(result.FailedInstances, inst.ID)
            consecutiveFailures++

            if consecutiveFailures >= d.MaxFailed {
                result.Status = "aborted"
                return result, fmt.Errorf(
                    "rolling update aborted: %d consecutive failures", consecutiveFailures)
            }
            continue
        }

        // Verify health
        inst.Version = newVersion
        if err := d.Health.Check(inst); err != nil {
            result.FailedInstances = append(result.FailedInstances, inst.ID)
            consecutiveFailures++

            if consecutiveFailures >= d.MaxFailed {
                result.Status = "aborted"
                return result, fmt.Errorf(
                    "rolling update aborted: health check failed on %s", inst.ID)
            }
            continue
        }

        result.UpdatedInstances = append(result.UpdatedInstances, inst.ID)
        consecutiveFailures = 0 // reset on success
        log.Printf("  ✓ %s healthy on %s", inst.ID, newVersion)
        // Output:   ✓ web-01 healthy on v1.3.0
    }

    result.Status = "complete"
    return result, nil
}

type DeployResult struct {
    Strategy         string
    NewVersion       string
    Status           string // complete, aborted, rolled_back
    UpdatedInstances []string
    FailedInstances  []string
}

The DeployAction function type lets callers inject their own deployment mechanism: restarting a Docker container, updating a Kubernetes pod, or SSH-ing to a VM.

Blue-Green Switching

Blue-green deployments maintain two identical environments. Only one receives traffic at a time:

package deploy

import (
    "context"
    "fmt"
    "log"
)

type Environment struct {
    Name      string     // "blue" or "green"
    Instances []Instance
    Active    bool
}

type BlueGreenDeployer struct {
    Health   *HealthChecker
    SwitchFn func(target string) error // switches load balancer
}

func (d *BlueGreenDeployer) Deploy(
    ctx context.Context,
    blue, green *Environment,
    newVersion string,
    deployFn DeployAction,
) error {
    // Determine which environment is inactive
    var target, active *Environment
    if blue.Active {
        active = blue
        target = green
    } else {
        active = green
        target = blue
    }

    log.Printf("Deploying %s to inactive environment: %s", newVersion, target.Name)
    // Output: Deploying v1.3.0 to inactive environment: green

    // Deploy to all instances in the inactive environment
    for _, inst := range target.Instances {
        if err := deployFn(ctx, inst, newVersion); err != nil {
            return fmt.Errorf("deploy to %s/%s: %w", target.Name, inst.ID, err)
        }
    }

    // Health check all target instances
    for _, inst := range target.Instances {
        inst.Version = newVersion
        if err := d.Health.Check(inst); err != nil {
            return fmt.Errorf("health check %s/%s: %w", target.Name, inst.ID, err)
        }
    }

    // Atomic switch
    log.Printf("Switching traffic: %s → %s", active.Name, target.Name)
    // Output: Switching traffic: blue → green
    if err := d.SwitchFn(target.Name); err != nil {
        return fmt.Errorf("switch traffic: %w", err)
    }

    target.Active = true
    active.Active = false

    log.Printf("✓ Blue-green deploy complete. Active: %s", target.Name)
    // Output: ✓ Blue-green deploy complete. Active: green
    return nil
}

Rollback is instant: call SwitchFn with the previous environment name.

Canary Deployment

Canary deployments route a small percentage of traffic to the new version, gradually increasing if metrics look good:

package deploy

import (
    "context"
    "fmt"
    "log"
    "time"
)

type CanaryConfig struct {
    Steps       []int         // traffic percentages: [5, 25, 50, 100]
    StepWait    time.Duration // observation period between steps
    MetricCheck func(version string) (healthy bool, err error)
}

type TrafficRouter interface {
    SetWeight(version string, percent int) error
}

func CanaryDeploy(
    ctx context.Context,
    router TrafficRouter,
    config CanaryConfig,
    newVersion string,
) error {
    for _, pct := range config.Steps {
        log.Printf("Canary: routing %d%% traffic to %s", pct, newVersion)
        // Output: Canary: routing 5% traffic to v1.3.0

        if err := router.SetWeight(newVersion, pct); err != nil {
            return fmt.Errorf("set weight %d%%: %w", pct, err)
        }

        // Wait and observe
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(config.StepWait):
        }

        // Check metrics
        healthy, err := config.MetricCheck(newVersion)
        if err != nil {
            // Rollback on metric check error
            _ = router.SetWeight(newVersion, 0)
            return fmt.Errorf("metric check at %d%%: %w", pct, err)
        }
        if !healthy {
            log.Printf("Canary unhealthy at %d%%, rolling back", pct)
            _ = router.SetWeight(newVersion, 0)
            return fmt.Errorf("canary failed at %d%% traffic", pct)
        }

        log.Printf("  ✓ Metrics healthy at %d%%", pct)
        // Output:   ✓ Metrics healthy at 5%
    }

    log.Printf("✓ Canary complete: %s receiving 100%% traffic", newVersion)
    // Output: ✓ Canary complete: v1.3.0 receiving 100% traffic
    return nil
}

Rollback Logic

A unified rollback controller wraps any deployment strategy:

package deploy

import (
    "context"
    "fmt"
    "log"
)

type Rollback struct {
    PreviousVersion string
    DeployFn        DeployAction
    Instances       []Instance
    Health          *HealthChecker
}

func (r *Rollback) Execute(ctx context.Context) error {
    log.Printf("ROLLBACK: reverting to %s", r.PreviousVersion)
    // Output: ROLLBACK: reverting to v1.2.0

    for _, inst := range r.Instances {
        if err := r.DeployFn(ctx, inst, r.PreviousVersion); err != nil {
            return fmt.Errorf("rollback %s: %w", inst.ID, err)
        }

        inst.Version = r.PreviousVersion
        if err := r.Health.Check(inst); err != nil {
            return fmt.Errorf("rollback health check %s: %w", inst.ID, err)
        }
    }

    log.Printf("✓ Rollback complete: all instances on %s", r.PreviousVersion)
    // Output: ✓ Rollback complete: all instances on v1.2.0
    return nil
}

Rolling Deploy Controller

A unified controller that performs a rolling deploy across a fleet of instances with automatic rollback on failure. It updates one instance at a time, verifies health after each, and if consecutive failures exceed a threshold, rolls back only the instances that were modified.

Input: A target version string (e.g., v1.3.0) passed as a CLI argument, with instance addresses and a deploy function configured in code.

Output: Deploys the new version instance-by-instance with health verification, automatically rolls back on failure, and exits with code 0 on success or 1 on failure for CI integration.

Full source: examples/rolling-deploy-controller

Key Takeaways

Rolling updates replace instances one-by-one with health verification between each, minimizing blast radius
Blue-green deployments pre-validate the entire inactive environment before an atomic traffic switch
Canary deployments use progressive traffic shifting with metric-based gates at each step
All strategies share the same rollback primitive: re-deploy the previous version and verify health
The DeployAction function type decouples strategy logic from the actual deployment mechanism (Docker, K8s, SSH)
Consecutive failure thresholds prevent rolling updates from degrading the entire fleet
Automatic rollback should only target instances that were actually modified, not the full set
Exit codes matter. Deployment tools should return non-zero on failure for CI pipeline integration