09 - Secrets & Vault Integration

📋 Jump to Takeaways

🎁 What if your application could get fresh, unique database credentials every time it starts, and no human ever sees or stores those passwords?

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Secret Management Patterns

Three common approaches for delivering secrets to applications, each with different security trade-offs:

Pattern	Mechanism	Pros	Cons
Env vars	`os.Getenv("DB_PASSWORD")`	Simple, universal	Visible in process lists, inherited by children
Files	Read from `/run/secrets/db`	Works with Docker secrets, tmpfs	File permissions critical, stale after rotation
API	Fetch from Vault at startup	Dynamic, auditable, rotatable	Network dependency, needs auth bootstrap

In production, the API pattern wins. Vault provides centralized access control, audit logging, dynamic credential generation, and automatic rotation. None of which environment variables or files offer.

HashiCorp Vault Client

The official Go client lives at github.com/hashicorp/vault/api:

go get github.com/hashicorp/vault/api

Create a client that connects to Vault and authenticates with a token:

import (
    "context"
    "fmt"

    vault "github.com/hashicorp/vault/api"
)

func newVaultClient(addr, token string) (*vault.Client, error) {
    config := vault.DefaultConfig()
    config.Address = addr

    client, err := vault.NewClient(config)
    if err != nil {
        return nil, fmt.Errorf("vault client init: %w", err)
    }
    client.SetToken(token)
    fmt.Printf("connected to vault at %s\n", addr)
    // Output: connected to vault at http://127.0.0.1:8200
    return client, nil
}

The client handles connection pooling, retries, and TLS automatically. In production, never hard-code the token. Use one of the auth methods below.

Authentication Methods

Token Auth (Development Only)

client.SetToken(os.Getenv("VAULT_TOKEN"))

Tokens expire. Never embed them in code or config files committed to Git.

AppRole Auth (Services & CI/CD)

AppRole is designed for machine-to-machine authentication. Your service gets a role_id (like a username) and a secret_id (like a password).

func loginAppRole(client *vault.Client, roleID, secretID string) error {
    data := map[string]interface{}{
        "role_id":   roleID,
        "secret_id": secretID,
    }

    resp, err := client.Logical().Write("auth/approle/login", data)
    if err != nil {
        return fmt.Errorf("approle login: %w", err)
    }

    client.SetToken(resp.Auth.ClientToken)
    fmt.Printf("approle login successful, token expires in %ds\n", resp.Auth.LeaseDuration)
    // Output: approle login successful, token expires in 3600s
    return nil
}

Kubernetes Auth (Pods)

When running in Kubernetes, pods authenticate using their service account token. No external secrets needed.

func loginKubernetes(client *vault.Client, role string) error {
    // K8s mounts the SA token at this path
    jwt, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token")
    if err != nil {
        return fmt.Errorf("reading SA token: %w", err)
    }

    data := map[string]interface{}{
        "jwt":  string(jwt),
        "role": role,
    }

    resp, err := client.Logical().Write("auth/kubernetes/login", data)
    if err != nil {
        return fmt.Errorf("k8s auth: %w", err)
    }

    client.SetToken(resp.Auth.ClientToken)
    return nil
}

This is the cleanest pattern for K8s. No secret distribution, no rotation headache. Vault validates the JWT against the cluster's token reviewer API.

Reading & Writing Secrets

KV v2 (Versioned Secrets)

func readSecret(client *vault.Client, path string) (map[string]interface{}, error) {
    // KV v2 prefixes the actual path with "data/"
    secret, err := client.KVv2("secret").Get(context.Background(), path)
    if err != nil {
        return nil, fmt.Errorf("read %s: %w", path, err)
    }
    fmt.Printf("read secret at %s (%d keys)\n", path, len(secret.Data))
    // Output: read secret at myapp/db (3 keys)
    return secret.Data, nil
}

func writeSecret(client *vault.Client, path string, data map[string]interface{}) error {
    _, err := client.KVv2("secret").Put(context.Background(), path, data)
    if err != nil {
        return fmt.Errorf("write %s: %w", path, err)
    }
    return nil
}

Batch Read (Multiple Paths)

type SecretSet struct {
    DB    map[string]interface{}
    Redis map[string]interface{}
    AWS   map[string]interface{}
}

func readAllSecrets(client *vault.Client) (*SecretSet, error) {
    set := &SecretSet{}

    type secretTarget struct {
        path   string
        target *map[string]interface{}
    }

    targets := []secretTarget{
        {"myapp/db", &set.DB},
        {"myapp/redis", &set.Redis},
        {"myapp/aws", &set.AWS},
    }

    for _, t := range targets {
        data, err := readSecret(client, t.path)
        if err != nil {
            return nil, err
        }
        *t.target = data
    }
    return set, nil
}

Dynamic Secrets & Leases

Dynamic secrets are generated on-demand and automatically revoked after a TTL. Vault creates unique database credentials per request, so there are no shared passwords.

func getDatabaseCreds(client *vault.Client, role string) (string, string, time.Duration, error) {
    secret, err := client.Logical().Read(fmt.Sprintf("database/creds/%s", role))
    if err != nil {
        return "", "", 0, fmt.Errorf("db creds: %w", err)
    }

    username := secret.Data["username"].(string)
    password := secret.Data["password"].(string)
    ttl := time.Duration(secret.LeaseDuration) * time.Second

    fmt.Printf("generated dynamic creds: user=%s ttl=%s\n", username, ttl)
    // Output: generated dynamic creds: user=v-approle-myapp-abc123 ttl=1h0m0s
    return username, password, ttl, nil
}

// Renew lease before it expires
func renewLease(client *vault.Client, leaseID string, increment int) error {
    _, err := client.Sys().Renew(leaseID, increment)
    return err
}

// Background lease renewal goroutine
func keepLeaseAlive(ctx context.Context, client *vault.Client, leaseID string, ttl time.Duration) {
    // Renew at 2/3 of TTL
    ticker := time.NewTicker(ttl * 2 / 3)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            // Revoke on shutdown
            client.Sys().Revoke(leaseID)
            return
        case <-ticker.C:
            if err := renewLease(client, leaseID, int(ttl.Seconds())); err != nil {
                fmt.Printf("lease renewal failed: %v\n", err)
                // Output: lease renewal failed: permission denied
                return
            }
            fmt.Printf("lease renewed: %s\n", leaseID)
            // Output: lease renewed: database/creds/myapp/abc123
        }
    }
}

Environment Injection

A common pattern: fetch secrets from Vault and inject them into environment variables before the application starts, bridging the API pattern with apps that expect env vars.

func injectEnv(client *vault.Client, mappings map[string]string) error {
    // mappings: vault path → env var name
    // e.g., "myapp/db#password" → "DB_PASSWORD"
    for vaultRef, envVar := range mappings {
        parts := strings.SplitN(vaultRef, "#", 2)
        path, key := parts[0], parts[1]

        data, err := readSecret(client, path)
        if err != nil {
            return fmt.Errorf("inject %s: %w", envVar, err)
        }

        value, ok := data[key].(string)
        if !ok {
            return fmt.Errorf("key %s not found in %s", key, path)
        }

        os.Setenv(envVar, value)
        fmt.Printf("injected %s from %s\n", envVar, path)
        // Output: injected DB_PASSWORD from myapp/db
    }
    return nil
}

Secret Rotation

Rotation means replacing secrets on a schedule without downtime. The pattern: fetch new credentials, verify connectivity, swap the active connection, revoke the old credentials.

type RotatingDB struct {
    mu     sync.RWMutex
    db     *sql.DB
    client *vault.Client
    role   string
}

func (r *RotatingDB) Rotate(ctx context.Context) error {
    // Get fresh credentials
    user, pass, ttl, err := getDatabaseCreds(r.client, r.role)
    if err != nil {
        return err
    }

    // Open new connection
    dsn := fmt.Sprintf("postgres://%s:%s@db-host:5432/myapp?sslmode=require", user, pass)
    newDB, err := sql.Open("postgres", dsn)
    if err != nil {
        return err
    }

    // Verify it works
    if err := newDB.PingContext(ctx); err != nil {
        newDB.Close()
        return fmt.Errorf("new creds failed ping: %w", err)
    }

    // Swap atomically
    r.mu.Lock()
    oldDB := r.db
    r.db = newDB
    r.mu.Unlock()

    // Close old connection (drain gracefully)
    if oldDB != nil {
        oldDB.Close()
    }

    // Schedule next rotation before TTL expires
    go func() {
        timer := time.NewTimer(ttl * 2 / 3)
        defer timer.Stop()
        select {
        case <-ctx.Done():
        case <-timer.C:
            r.Rotate(ctx)
        }
    }()

    return nil
}

func (r *RotatingDB) Query(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error) {
    r.mu.RLock()
    db := r.db
    r.mu.RUnlock()
    return db.QueryContext(ctx, query, args...)
}

Vault Config Loader

A complete application that authenticates to Vault (auto-detecting Kubernetes vs AppRole environment), reads secrets from KV v2, and populates a typed config struct for the application to use.

Input: Vault address and auth credentials (AppRole role/secret IDs or Kubernetes service account). Output: A populated AppConfig struct with database, Redis, and API credentials ready for use.

Full source: examples/vault-config-loader

Key Takeaways

Prefer API-based secret delivery (Vault) over env vars or files for auditability and rotation.
Use AppRole for CI/CD pipelines and Kubernetes auth for pods. Both avoid distributing long-lived tokens.
Dynamic secrets (database/creds) give each service instance unique credentials that auto-expire.
Implement lease renewal goroutines that renew at 2/3 TTL and revoke on shutdown.
Secret rotation swaps connections atomically under a sync.RWMutex, so readers never block.
Structure your auth logic to auto-detect the environment (K8s vs CI) for portable services.

🎁 How do you know if your service is slow, overloaded, or about to fall over, before users start complaining?