14 - Deployment Automation

📋 Jump to Takeaways

🎁 What if you could replace every running instance of your app with a new version, and not a single user notices anything happened?

You push a bad deploy. Five minutes later your error rate is 40%. You run kubectl set image deployment/api api=myregistry/api:v1.2.1 to roll back. But kubectl is not in your PATH on this machine.

You find it, run the command. It hangs. The deployment is stuck because a pod can't schedule. The old pods are terminating. The new pods are pending. Nothing is serving traffic.

Twenty minutes of chaos before service is restored, and only because someone knew to run kubectl rollout undo.

Rolling updates without automated health verification and rollback are a manual process. They rely on humans being alert and fast at 2am. The goal of deployment automation is to make rollbacks automatic, immediate, and surgical. When a deploy starts failing, the system rolls back without anyone having to notice.

This lesson adds devctl rollout deploy <service> <version> to the tool — a command that updates instances one by one, verifies health after each, and rolls back automatically if the threshold is crossed.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Why Not Just Re-deploy the Old Version?

When a rolling update goes wrong, the first instinct is to deploy the old version to all instances. That's wrong in one specific scenario: partial rollouts.

If your deploy updated 3 of 8 instances before failing, you should roll back those 3. Rolling back all 8 means restarting 5 already-healthy instances for no reason. At 2am with degraded service, every unnecessary restart is another risk of something else going wrong.

Automated rollback needs to track which instances were actually touched and target only those. The deployer records UpdatedInstances as it goes. If it aborts, the rollback list is exactly what was modified.

Deployment Strategies Overview

Three strategies control how traffic shifts from old to new versions. All three share a core principle: never commit fully to a new version until health is verified.

Strategy	Mechanism	Risk	Rollback Speed
Rolling	Replace instances one-by-one	Gradual	Moderate (re-deploy old)
Blue-Green	Run two identical environments, switch atomically	Low (full pre-validation)	Instant (switch back)
Canary	Route small % of traffic to new version	Very low	Instant (route 0%)

Instance and Health Check Model

Start with the types that represent your deployment targets. DeployAction is a function, not a method, so you can inject any mechanism: restarting a Docker container, calling the Kubernetes API, or SSH-ing to a VM. The deployer doesn't care how the instance gets updated.

package deploy

import (
    "context"
    "fmt"
    "net/http"
    "time"
)

type Instance struct {
    ID       string
    Address  string
    Version  string
    Healthy  bool
}

type HealthChecker struct {
    Client   *http.Client
    Endpoint string // e.g., "/healthz"
    Retries  int
    Interval time.Duration
}

func (h *HealthChecker) Check(ctx context.Context, address string) error {
    url := fmt.Sprintf("http://%s%s", address, h.Endpoint)

    for attempt := 0; attempt <= h.Retries; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return err
        }
        resp, err := h.Client.Do(req)
        if err == nil && resp.StatusCode == http.StatusOK {
            resp.Body.Close()
            return nil
        }
        if resp != nil {
            resp.Body.Close()
        }

        if attempt < h.Retries {
            time.Sleep(h.Interval)
        }
    }
    return fmt.Errorf("address %s unhealthy after %d attempts", address, h.Retries+1)
}

Rolling Update Implementation

Rolling updates replace instances one at a time, verifying health after each replacement. If health checks fail consecutively, the update aborts and only the already-updated instances get rolled back.

package deploy

import (
    "context"
    "fmt"
    "log"
)

type DeployResult struct {
    Strategy         string
    NewVersion       string
    Status           string // complete, aborted, rolled_back
    UpdatedInstances []string
    FailedInstances  []string
}

type RollingDeployer struct {
    Health    *HealthChecker
    MaxFailed int // max consecutive failures before abort
}

type DeployAction func(ctx context.Context, instance Instance, newVersion string) error

func (d *RollingDeployer) Deploy(
    ctx context.Context,
    instances []Instance,
    newVersion string,
    deployFn DeployAction,
) (*DeployResult, error) {
    result := &DeployResult{
        Strategy:   "rolling",
        NewVersion: newVersion,
    }

    consecutiveFailures := 0

    for i, inst := range instances {
        log.Printf("[%d/%d] Deploying %s: %s -> %s",
            i+1, len(instances), inst.ID, inst.Version, newVersion)
        // Output: [1/4] Deploying web-01: v1.2.0 -> v1.3.0

        if err := deployFn(ctx, inst, newVersion); err != nil {
            result.FailedInstances = append(result.FailedInstances, inst.ID)
            consecutiveFailures++

            if consecutiveFailures >= d.MaxFailed {
                result.Status = "aborted"
                return result, fmt.Errorf(
                    "rolling update aborted: %d consecutive failures", consecutiveFailures)
            }
            continue
        }

        // deployFn succeeded: record as modified before health check.
        // If health check fails, the new version is still running and must be rolled back.
        result.UpdatedInstances = append(result.UpdatedInstances, inst.ID)

        if err := d.Health.Check(ctx, inst.Address); err != nil {
            result.FailedInstances = append(result.FailedInstances, inst.ID)
            consecutiveFailures++

            if consecutiveFailures >= d.MaxFailed {
                result.Status = "aborted"
                return result, fmt.Errorf(
                    "rolling update aborted: health check failed on %s", inst.ID)
            }
            continue
        }

        consecutiveFailures = 0 // reset on success
        log.Printf("  + %s healthy on %s", inst.ID, newVersion)
        // Output:   + web-01 healthy on v1.3.0
    }

    result.Status = "complete"
    return result, nil
}

result.UpdatedInstances is the key field. When the deploy aborts, this is exactly the list that needs to be rolled back. Not all instances. Not all instances minus the failed ones. Exactly the instances where deployFn succeeded, because those are running the new version.

Blue-Green Switching

Blue-green deployments maintain two identical environments. Only one receives traffic at a time. The inactive environment gets updated and fully health-checked before the traffic switch. If the switch fails, you call SwitchFn with the previous environment name. Rollback is one function call.

package deploy

import (
    "context"
    "fmt"
    "log"
)

type Environment struct {
    Name      string     // "blue" or "green"
    Instances []Instance
    Active    bool
}

type BlueGreenDeployer struct {
    Health   *HealthChecker
    SwitchFn func(target string) error // switches load balancer
}

func (d *BlueGreenDeployer) Deploy(
    ctx context.Context,
    blue, green *Environment,
    newVersion string,
    deployFn DeployAction,
) error {
    var target, active *Environment
    if blue.Active {
        active = blue
        target = green
    } else {
        active = green
        target = blue
    }

    log.Printf("Deploying %s to inactive environment: %s", newVersion, target.Name)
    // Output: Deploying v1.3.0 to inactive environment: green

    for _, inst := range target.Instances {
        if err := deployFn(ctx, inst, newVersion); err != nil {
            return fmt.Errorf("deploy to %s/%s: %w", target.Name, inst.ID, err)
        }
    }

    for _, inst := range target.Instances {
        if err := d.Health.Check(ctx, inst.Address); err != nil {
            return fmt.Errorf("health check %s/%s: %w", target.Name, inst.ID, err)
        }
    }

    log.Printf("Switching traffic: %s -> %s", active.Name, target.Name)
    // Output: Switching traffic: blue -> green
    if err := d.SwitchFn(target.Name); err != nil {
        return fmt.Errorf("switch traffic: %w", err)
    }

    target.Active = true
    active.Active = false

    log.Printf("Blue-green deploy complete. Active: %s", target.Name)
    // Output: Blue-green deploy complete. Active: green
    return nil
}

Canary Deployment

Canary deployments route a small percentage of traffic to the new version, gradually increasing if metrics look good. At each step, you observe error rates, latency, or any metric that matters. A bad metric triggers an immediate rollback: set the weight to 0%.

package deploy

import (
    "context"
    "fmt"
    "log"
    "time"
)

type CanaryConfig struct {
    Steps       []int         // traffic percentages: [5, 25, 50, 100]
    StepWait    time.Duration // observation period between steps
    MetricCheck func(version string) (healthy bool, err error)
}

type TrafficRouter interface {
    SetWeight(version string, percent int) error
}

func CanaryDeploy(
    ctx context.Context,
    router TrafficRouter,
    config CanaryConfig,
    newVersion string,
) error {
    for _, pct := range config.Steps {
        log.Printf("Canary: routing %d%% traffic to %s", pct, newVersion)
        // Output: Canary: routing 5% traffic to v1.3.0

        if err := router.SetWeight(newVersion, pct); err != nil {
            return fmt.Errorf("set weight %d%%: %w", pct, err)
        }

        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(config.StepWait):
        }

        healthy, err := config.MetricCheck(newVersion)
        if err != nil {
            if wErr := router.SetWeight(newVersion, 0); wErr != nil {
                log.Printf("canary: failed to reset weight: %v", wErr)
            }
            return fmt.Errorf("metric check at %d%%: %w", pct, err)
        }
        if !healthy {
            log.Printf("Canary unhealthy at %d%%, rolling back", pct)
            if wErr := router.SetWeight(newVersion, 0); wErr != nil {
                log.Printf("canary: failed to reset weight: %v", wErr)
            }
            return fmt.Errorf("canary failed at %d%% traffic", pct)
        }

        log.Printf("  + Metrics healthy at %d%%", pct)
        // Output:   + Metrics healthy at 5%
    }

    log.Printf("Canary complete: %s receiving 100%% traffic", newVersion)
    // Output: Canary complete: v1.3.0 receiving 100% traffic
    return nil
}

Rollback Logic

A unified rollback controller takes the list of instances that were actually modified and redeploys the previous version to only those. This is the surgical rollback: no healthy instances are touched.

package deploy

import (
    "context"
    "fmt"
    "log"
)

type Rollback struct {
    PreviousVersion   string
    DeployFn          DeployAction
    ModifiedInstances []Instance // only instances that received the new version
    Health            *HealthChecker
}

func (r *Rollback) Execute(ctx context.Context) error {
    log.Printf("ROLLBACK: reverting %d instances to %s",
        len(r.ModifiedInstances), r.PreviousVersion)
    // Output: ROLLBACK: reverting 3 instances to v1.2.0

    for _, inst := range r.ModifiedInstances {
        if err := r.DeployFn(ctx, inst, r.PreviousVersion); err != nil {
            return fmt.Errorf("rollback %s: %w", inst.ID, err)
        }

        if err := r.Health.Check(ctx, inst.Address); err != nil {
            return fmt.Errorf("rollback health check %s: %w", inst.ID, err)
        }
    }

    log.Printf("Rollback complete: modified instances on %s", r.PreviousVersion)
    // Output: Rollback complete: modified instances on v1.2.0
    return nil
}

Wire the deployer and rollback together. filterInstances converts the ID strings in result.UpdatedInstances back into Instance structs:

func filterInstances(all []Instance, ids []string) []Instance {
    set := make(map[string]bool, len(ids))
    for _, id := range ids {
        set[id] = true
    }
    var out []Instance
    for _, inst := range all {
        if set[inst.ID] {
            out = append(out, inst)
        }
    }
    return out
}

When the rolling deploy aborts, pass result.UpdatedInstances to Rollback.ModifiedInstances:

result, err := deployer.Deploy(ctx, instances, newVersion, deployFn)
if err != nil {
    modifiedInstances := filterInstances(instances, result.UpdatedInstances)
    rb := &Rollback{
        PreviousVersion:   previousVersion,
        DeployFn:          deployFn,
        ModifiedInstances: modifiedInstances,
        Health:            health,
    }
    if rbErr := rb.Execute(ctx); rbErr != nil {
        log.Printf("Rollback also failed: %v", rbErr)
    }
}

If 3 of 8 instances were updated before failure, only those 3 get the rollback. The other 5 never stopped serving traffic on the old version.

Putting It Together: the Rollout Command

Wire the deployer as the devctl rollout deploy subcommand:

func newRolloutDeployCmd() *cobra.Command {
    cmd := &cobra.Command{
        Use:   "deploy <service> <version>",
        Short: "Rolling deploy with automatic rollback",
        Args:  cobra.ExactArgs(2),
        RunE: func(cmd *cobra.Command, args []string) error {
            return runRollingDeploy(cmd.Context(), args[0], args[1])
        },
    }
    return cmd
}

var rolloutCmd = &cobra.Command{Use: "rollout", Short: "Deployment commands"}

func init() {
    rolloutCmd.AddCommand(newRolloutDeployCmd())
    rootCmd.AddCommand(rolloutCmd)
}

Go back to the intro scenario. Your deploy updates 3 of 8 instances. Health checks start failing. With automated rollback in the loop:

The deployer records each successful update in UpdatedInstances. After the second consecutive failure, it aborts and returns the result. The orchestrator sees a non-zero error. It creates a Rollback with ModifiedInstances set to the 3 instances in UpdatedInstances. The rollback runs. Three instances get the previous version. Health checks pass. Done.

Total human involvement: zero. Total time from failure to recovery: however long it takes to deploy to 3 instances and check their health. Not 20 minutes of someone SSHing around trying to find kubectl.

The only thing you need to get this right is tracking which instances were modified. Everything else follows from that.

func main() {
    newVersion := "v1.3.0"
    if len(os.Args) > 1 {
        newVersion = os.Args[1]
    }

    instances := []Instance{
        {ID: "web-01", Address: "10.0.1.1:8080", Version: "v1.2.0"},
        {ID: "web-02", Address: "10.0.1.2:8080", Version: "v1.2.0"},
        {ID: "web-03", Address: "10.0.1.3:8080", Version: "v1.2.0"},
        {ID: "web-04", Address: "10.0.1.4:8080", Version: "v1.2.0"},
    }
    previousVersion := instances[0].Version

    health := &HealthChecker{
        Client:   &http.Client{Timeout: 5 * time.Second},
        Endpoint: "/healthz",
        Retries:  3,
        Interval: 2 * time.Second,
    }

    deployer := &RollingDeployer{Health: health, MaxFailed: 2}

    deployFn := func(ctx context.Context, inst Instance, version string) error {
        log.Printf("updating %s to %s", inst.ID, version)
        return nil // replace with real Docker/K8s call
    }

    result, err := deployer.Deploy(context.Background(), instances, newVersion, deployFn)
    if err != nil {
        log.Printf("deploy failed: %v, rolling back %d instances", err, len(result.UpdatedInstances))

        modifiedInstances := filterInstances(instances, result.UpdatedInstances)
        rb := &Rollback{
            PreviousVersion:   previousVersion,
            DeployFn:          deployFn,
            ModifiedInstances: modifiedInstances,
            Health:            health,
        }
        if rbErr := rb.Execute(context.Background()); rbErr != nil {
            log.Fatalf("rollback also failed: %v", rbErr)
        }
        os.Exit(1)
    }

    fmt.Printf("deployed %s to %d instances\n", newVersion, len(result.UpdatedInstances))
}

Rolling Deploy Controller

A unified controller that performs a rolling deploy across a fleet of instances with automatic rollback on failure. It updates one instance at a time, verifies health after each, and if consecutive failures exceed a threshold, rolls back only the instances that were modified.

Input: A target version string (e.g., v1.3.0) passed as a CLI argument, with instance addresses and a deploy function configured in code.

Output: Deploys the new version instance-by-instance with health verification, automatically rolls back modified instances on failure, and exits with code 0 on success or 1 on failure for CI integration.

Full source: examples/rolling-deploy-controller

Key Takeaways

Rolling updates replace instances one-by-one with health verification between each. A failure threshold aborts the update before the entire fleet is degraded.
Track which instances were actually modified. Rollback targets only those instances, not the full fleet.
Record UpdatedInstances as soon as deployFn succeeds, before the health check. If health check fails, the new version is still running and must be rolled back.
Blue-green deployments pre-validate the entire inactive environment before an atomic traffic switch. Rollback is one load balancer call.
Canary deployments use progressive traffic shifting with metric-based gates at each step. Rollback is setting the weight to 0%.
The DeployAction function type decouples strategy logic from the actual deployment mechanism (Docker, Kubernetes, SSH).
Automated rollback needs the previous version recorded before the deploy starts. Don't assume you can look it up later.
Exit codes matter. Deployment tools should return non-zero on failure for CI pipeline integration.

🎁 devctl rollout deploy now handles your deploys. The last piece: how do you get the devctl binary itself onto 40 engineers' machines and make sure nobody is running a stale version?