09 - Secrets & Vault Integration

🎁 What if your application could get fresh, unique database credentials every time it starts, and no human ever sees or stores those passwords?

This lesson adds devctl secrets load to the CLI. It authenticates to Vault, fetches the secrets a service needs, and injects them as environment variables before handing off to the app.

You're doing a security audit. You grep your repos for secrets. What you find is not the result of carelessness, it's the result of people moving fast and picking the simplest thing that worked.

A database password in a .env file, committed two years ago. Still valid. An API key hardcoded in config.yaml, one of them for a production payment processor. A Slack bot token in a CI script, world-readable to anyone with repo access. None of this was malicious. Nobody sat down and decided to make the company insecure. They were shipping.

The question isn't how to blame people for this. It's how to build a system where the easy path is also the secure one.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

The Bootstrap Problem (and Why It's Worth Solving)

Every secrets solution has a bootstrap problem: to get a secret, you need a credential. That credential has to come from somewhere. You can't fully eliminate it.

What you can do is make it much less dangerous. That's the real argument for Vault.

Hardcoded strings and env vars store the actual secret — the DB password, the API key — directly. Leak the config and you've leaked the secret. No audit trail, no way to revoke it without a rotation, no way to scope it to one service.

Files on disk are one step better. Docker secrets and Kubernetes volume mounts keep secrets out of source code. But they still store the raw credential. The file gets accidentally committed during development, goes stale after rotation, gets copied to someone's laptop for debugging.

Vault doesn't eliminate the bootstrap credential — but it changes what that credential can do. An AppRole secret_id is not the database password. It's only the right to ask Vault for it. Compromising a secret_id doesn't expose your production database directly — it exposes a narrowly scoped, short-lived, revocable token that Vault audits on every use.

The Kubernetes auth method goes further and eliminates the bootstrap credential entirely. A pod authenticates to Vault using its Kubernetes service account JWT — a token Kubernetes issues automatically. There's nothing to distribute, rotate, or accidentally commit. The pod's identity is its credential.

Here's the comparison that actually matters:

The goal isn't "no credentials." It's "credentials with limited blast radius that expire automatically and leave an audit trail." Vault delivers that.

go get github.com/hashicorp/vault/api

Secret Management Patterns

The three approaches, with their real trade-offs:

¹ /run/secrets/db is the path where Docker writes a secret named db when you mount it into a container with --secret. Kubernetes uses the same pattern with volume mounts — you pick the mountPath and each key in the Secret becomes a file at that path. Your app reads it with os.ReadFile("/run/secrets/db"). The path is not a fixed standard; it depends on how you configure the mount.

For production, the API pattern wins. Vault provides centralized access control, audit logging, dynamic credential generation, and automatic rotation. None of which env vars or files offer.

HashiCorp Vault Client

You could call Vault's REST API with net/http. The better approach is using the SDK, which adds token renewal (Vault tokens expire), lease management for dynamic credentials, KV v2's data/ path convention (easy to get wrong manually), and typed response structs for each secrets engine.

Create a client and connect to Vault:

import (
    "context"
    "fmt"

    vault "github.com/hashicorp/vault/api"
)

func newVaultClient(addr, token string) (*vault.Client, error) {
    config := vault.DefaultConfig()
    config.Address = addr

    client, err := vault.NewClient(config)
    if err != nil {
        return nil, fmt.Errorf("vault client init: %w", err)
    }
    client.SetToken(token)
    fmt.Printf("connected to vault at %s\n", addr)
    // Output: connected to vault at http://127.0.0.1:8200
    return client, nil
}

The client handles connection pooling, retries, and TLS. In production, never pass a hard-coded token to this function. Use one of the auth methods below.

Authentication Methods

Token Auth (Development Only)

client.SetToken(os.Getenv("VAULT_TOKEN"))

Tokens expire. Fine for local development. Not for production.

Gotcha: Don't put VAULT_TOKEN in a Kubernetes manifest as an env var. Even if you pull the value from a k8s Secret, you've now stored a long-lived Vault token inside Kubernetes, which creates its own rotation problem. The Kubernetes auth method below is the right answer for pods.

AppRole Auth (Services and CI/CD)

AppRole is designed for machine-to-machine authentication. Your service gets a role_id (like a username) and a secret_id (like a one-time password). Each CI job gets a fresh secret_id that expires after use.

func loginAppRole(client *vault.Client, roleID, secretID string) error {
    data := map[string]interface{}{
        "role_id":   roleID,
        "secret_id": secretID,
    }

    resp, err := client.Logical().Write("auth/approle/login", data)
    if err != nil {
        return fmt.Errorf("approle login: %w", err)
    }

    client.SetToken(resp.Auth.ClientToken)
    fmt.Printf("approle login successful, token expires in %ds\n", resp.Auth.LeaseDuration)
    // Output: approle login successful, token expires in 3600s
    return nil
}

Gotcha: LeaseDuration tells you the token TTL in seconds. If you authenticate at startup and store the token, it'll be invalid in an hour. Your service will start failing Vault reads with 403 permission denied and you'll spend time wondering why it worked for the first hour. The token renewal pattern below handles this.

Kubernetes Auth (Pods)

When running inside a Kubernetes pod, use the service account JWT instead — no credentials to distribute. Vault validates the JWT against the cluster's token reviewer API.

func loginKubernetes(client *vault.Client, role string) error {
    // Kubernetes mounts the service account token at this path
    jwt, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token")
    if err != nil {
        return fmt.Errorf("reading SA token: %w", err)
    }

    data := map[string]interface{}{
        "jwt":  string(jwt),
        "role": role,
    }

    resp, err := client.Logical().Write("auth/kubernetes/login", data)
    if err != nil {
        return fmt.Errorf("k8s auth: %w", err)
    }

    client.SetToken(resp.Auth.ClientToken)
    return nil
}

This is the cleanest pattern for pods. No secret distribution, no rotation headache. The pod's service account is the identity.

Token Renewal

After authentication, the token you have will expire. For a long-running service, you need to renew it before expiry. The pattern is a background goroutine that renews at 2/3 of the TTL.

func keepTokenAlive(ctx context.Context, client *vault.Client, ttl time.Duration) {
    // Renew at 2/3 of TTL to give yourself a buffer
    ticker := time.NewTicker(ttl * 2 / 3)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            secret, err := client.Auth().Token().RenewSelf(int(ttl.Seconds()))
            if err != nil {
                fmt.Printf("token renewal failed: %v — attempting re-auth\n", err)
                // Output: token renewal failed: permission denied — attempting re-auth
                // If renewal fails, re-authenticate from scratch
                return
            }
            fmt.Printf("vault token renewed, next renewal in %s\n", ttl*2/3)
            // Output: vault token renewed, next renewal in 40m0s
            _ = secret
        }
    }
}

Call this in a goroutine right after authentication. When the context is cancelled (service shutdown), the goroutine exits cleanly.

initVault wraps the full bootstrap: create the client, auto-detect the auth method, start the renewal goroutine:

// initVault creates a Vault client and authenticates it.
// In a Kubernetes pod (KUBERNETES_SERVICE_HOST is set), it uses the pod's
// service account JWT via k8sRole. Elsewhere it uses AppRole credentials.
func initVault(ctx context.Context, addr, roleID, secretID, k8sRole string) (*vault.Client, error) {
    client, err := newVaultClient(addr, "")
    if err != nil {
        return nil, err
    }

    // Auto-detect environment: Kubernetes pods have KUBERNETES_SERVICE_HOST set
    if os.Getenv("KUBERNETES_SERVICE_HOST") != "" {
        if err := loginKubernetes(client, k8sRole); err != nil {
            return nil, err
        }
    } else {
        if err := loginAppRole(client, roleID, secretID); err != nil {
            return nil, err
        }
    }

    ttl := time.Hour // match your Vault policy's token TTL
    go keepTokenAlive(ctx, client, ttl)

    return client, nil
}

Wire the full flow into devctl as the secrets load subcommand:

var secretsLoadCmd = &cobra.Command{
    Use:   "load",
    Short: "Fetch secrets from Vault and inject them into the environment",
    RunE: func(cmd *cobra.Command, args []string) error {
        ctx, cancel := context.WithCancel(cmd.Context())
        defer cancel()

        client, err := initVault(ctx,
            os.Getenv("VAULT_ADDR"),
            os.Getenv("VAULT_ROLE_ID"),
            os.Getenv("VAULT_SECRET_ID"),
            "myapp",
        )
        if err != nil {
            return fmt.Errorf("vault init: %w", err)
        }

        secrets, err := readAllSecrets(client)
        if err != nil {
            return fmt.Errorf("load secrets: %w", err)
        }

        fmt.Printf("loaded %d DB keys, %d Redis keys\n",
            len(secrets.DB), len(secrets.Redis))
        // Output: loaded 3 DB keys, 2 Redis keys
        return nil
    },
}

func init() {
    secretsCmd.AddCommand(secretsLoadCmd)
    rootCmd.AddCommand(secretsCmd)
}

Run it before starting any service:

devctl secrets load

Reading and Writing Secrets

KV v2 (Versioned Secrets)

func readSecret(client *vault.Client, path string) (map[string]interface{}, error) {
    // KV v2 prefixes the actual path with "data/" — the SDK handles this
    secret, err := client.KVv2("secret").Get(context.Background(), path)
    if err != nil {
        return nil, fmt.Errorf("read %s: %w", path, err)
    }
    fmt.Printf("read secret at %s (%d keys)\n", path, len(secret.Data))
    // Output: read secret at myapp/db (3 keys)
    return secret.Data, nil
}

Writing secrets uses the same path but calls client.KVv2("secret").Put(ctx, path, data) — useful for seeding test environments or rotating stored credentials.

Reading Multiple Paths at Startup

Most services need several secrets at startup. Read them in a loop before serving traffic:

type SecretSet struct {
    DB    map[string]interface{}
    Redis map[string]interface{}
    AWS   map[string]interface{}
}

func readAllSecrets(client *vault.Client) (*SecretSet, error) {
    set := &SecretSet{}

    type secretTarget struct {
        path   string
        target *map[string]interface{}
    }

    targets := []secretTarget{
        {"myapp/db", &set.DB},
        {"myapp/redis", &set.Redis},
        {"myapp/aws", &set.AWS},
    }

    for _, t := range targets {
        data, err := readSecret(client, t.path)
        if err != nil {
            return nil, err
        }
        *t.target = data
    }
    return set, nil
}

Dynamic Secrets and Lease Renewal

Dynamic secrets are generated on demand and automatically revoked after a TTL. Each service instance gets unique database credentials. No shared passwords, no credential reuse across services, no "which service is using this password?" after a breach.

func getDatabaseCreds(client *vault.Client, role string) (string, string, time.Duration, error) {
    secret, err := client.Logical().Read(fmt.Sprintf("database/creds/%s", role))
    if err != nil {
        return "", "", 0, fmt.Errorf("db creds: %w", err)
    }

    username, ok := secret.Data["username"].(string)
    if !ok {
        return "", "", 0, fmt.Errorf("unexpected type for username in vault response")
    }
    password, ok := secret.Data["password"].(string)
    if !ok {
        return "", "", 0, fmt.Errorf("unexpected type for password in vault response")
    }
    ttl := time.Duration(secret.LeaseDuration) * time.Second

    fmt.Printf("generated dynamic creds: user=%s ttl=%s\n", username, ttl)
    // Output: generated dynamic creds: user=v-approle-myapp-abc123 ttl=1h0m0s
    return username, password, ttl, nil
}

Dynamic credentials have a lease ID. Renew the lease before it expires, or Vault revokes the database credentials and your queries start failing. The rotation pattern in the next section handles this inline: RotatingDB.Rotate fetches fresh credentials and schedules the next rotation at 2/3 of the TTL, so the lease never goes stale.

Secret Rotation Without Downtime

For services that hold a live database connection, rotation means swapping credentials while queries are in flight. The pattern: fetch new credentials, verify connectivity, swap the active connection under a mutex, close the old connection.

type RotatingDB struct {
    mu     sync.RWMutex
    db     *sql.DB
    client *vault.Client
    role   string
}

func (r *RotatingDB) Rotate(ctx context.Context) error {
    user, pass, ttl, err := getDatabaseCreds(r.client, r.role)
    if err != nil {
        return err
    }

    dsn := fmt.Sprintf("postgres://%s:%s@db-host:5432/myapp?sslmode=require", user, pass)
    newDB, err := sql.Open("postgres", dsn)
    if err != nil {
        return err
    }

    if err := newDB.PingContext(ctx); err != nil {
        newDB.Close()
        return fmt.Errorf("new creds failed ping: %w", err)
    }

    // Swap atomically so in-flight readers keep the old connection
    r.mu.Lock()
    oldDB := r.db
    r.db = newDB
    r.mu.Unlock()

    if oldDB != nil {
        oldDB.Close()
    }

    // Schedule next rotation at 2/3 of TTL.
    // This goroutine fires once and exits. The next Rotate call schedules
    // a new one. Never call Rotate manually in a loop — each call spawns
    // a new timer goroutine.
    go func() {
        timer := time.NewTimer(ttl * 2 / 3)
        defer timer.Stop()
        select {
        case <-ctx.Done():
        case <-timer.C:
            r.Rotate(ctx)
        }
    }()

    return nil
}

func (r *RotatingDB) Query(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error) {
    r.mu.RLock()
    db := r.db
    r.mu.RUnlock()
    return db.QueryContext(ctx, query, args...)
}

Readers hold an RLock only for the pointer copy, not for the query itself. The rotation swap holds an exclusive lock for a microsecond. Queries are never blocked.

Environment Injection

Some applications expect secrets as env vars. A secrets loader can bridge the API pattern with those apps: fetch from Vault at startup, inject into the environment, then start the app.

func injectEnv(client *vault.Client, mappings map[string]string) error {
    // mappings: "myapp/db#password" -> "DB_PASSWORD"
    for vaultRef, envVar := range mappings {
        parts := strings.SplitN(vaultRef, "#", 2)
        if len(parts) != 2 {
            return fmt.Errorf("invalid vault ref %q: expected path#key format", vaultRef)
        }
        path, key := parts[0], parts[1]

        data, err := readSecret(client, path)
        if err != nil {
            return fmt.Errorf("inject %s: %w", envVar, err)
        }

        value, ok := data[key].(string)
        if !ok {
            return fmt.Errorf("key %s not found in %s", key, path)
        }

        os.Setenv(envVar, value)
        fmt.Printf("injected %s from %s\n", envVar, path)
        // Output: injected DB_PASSWORD from myapp/db
    }
    return nil
}

This is a bridge, not a goal. If you control the application, fetch secrets directly and pass them as typed config values. Use environment injection only for third-party apps that you can't modify.

Vault Config Loader

A complete application that authenticates to Vault (auto-detecting Kubernetes vs AppRole environment), reads secrets from KV v2, and populates a typed config struct for the application to use.

Input: Vault address and auth credentials (AppRole role/secret IDs or Kubernetes service account). Output: A populated AppConfig struct with database, Redis, and API credentials ready for use.

Full source: examples/vault-config-loader

Key Takeaways

• Env vars appear in ps aux output and get inherited by every child process. Files get committed accidentally. The API pattern solves both.
• Use AppRole for CI/CD pipelines and Kubernetes auth for pods. Both avoid distributing long-lived tokens.
• Vault tokens expire. Authenticate at startup, then run a background goroutine that renews at 2/3 of the TTL. If renewal fails, re-authenticate from scratch.
• Never put VAULT_TOKEN in a Kubernetes manifest as an env var. Use the Kubernetes auth method instead.
• Dynamic secrets (database/creds) give each service instance unique credentials that auto-expire. Each needs a lease renewal goroutine or Vault will revoke the credentials.
• Rotate live database connections by opening a new connection, verifying it, swapping under a sync.RWMutex, then closing the old one.
• Structure your auth logic to auto-detect the environment (Kubernetes vs CI) for portable services.

🎁 Secrets are loaded. The service is running. Now, how do you know it's healthy? Next, devctl metrics starts the Prometheus exporter so you get queue depth and latency alerts before users notice anything is wrong.