13 - CI/CD Pipeline Tooling

📋 Jump to Takeaways

🎁 GitHub Actions runs your pipeline, but what actually runs GitHub Actions? What if you could build the runner itself in under 200 lines of Go?

Your CI is GitHub Actions. It runs fine, until you need to debug a failing test. You can't run GitHub Actions locally. You push a commit, wait four minutes, read the logs, push another commit. Twelve iterations later you have twelve commits: "fix", "fix 2", "actually fix", "try different approach", "revert", "revert revert".

The fundamental problem: the pipeline is YAML that runs on someone else's machine. You have no way to execute a single step locally with the same environment and the same logic. When a step fails, your debugging loop is "commit, push, wait."

A pipeline runner you build yourself gives you something different. It's a binary that runs on your laptop exactly as it runs in CI. You can run a single step, inspect the output, change the code, and run it again. No commits, no waits.

This lesson builds that runner from scratch, and adds it to devctl as devctl pipeline run <pipeline.yaml>.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Why Not Run Pipeline Steps From a Bash Script?

A bash script can run your test and lint commands. That's not the hard part.

The hard part is timeouts. A runaway test can hang your entire pipeline. The hard part is artifact collection: copying specific output files to a named location after a successful build. The hard part is webhook reporting: posting structured build status to Slack or GitHub after each step. The hard part is cancellation: when you press Ctrl-C or the CI job is cancelled, you need all child processes to stop, not just the runner.

Shell scripts handle none of that cleanly. Go gives you typed data structures, context cancellation, and process management, and the binary runs on any platform without a dependency on bash.

Anatomy of a CI Runner

Every CI runner follows the same loop:

Poll for work (or receive a webhook trigger)
Parse the pipeline definition (YAML, JSON, or DSL)
Execute steps sequentially or in parallel as subprocesses
Capture output, exit codes, and artifacts
Report status back to the orchestrator or external system

The runner itself is stateless. Pipeline definitions declare everything. This is why CI systems are so portable: the runner is just an execution engine.

Pipeline Definition as a Go Struct

Define your pipeline schema as Go structs with YAML tags. A struct gives you typed access to every field and catches schema mistakes at parse time, not at step 7 of a 10-step pipeline.

package pipeline

import "time"

type Pipeline struct {
    Name      string   `yaml:"name"`
    Trigger   string   `yaml:"trigger"`
    Steps     []Step   `yaml:"steps"`
    Artifacts []string `yaml:"artifacts"`
}

type Step struct {
    Name    string   `yaml:"name"`
    Command string   `yaml:"command"`
    Timeout string   `yaml:"timeout"` // "30s", "2m", parsed with time.ParseDuration
    Env     []string `yaml:"env"`
}

A corresponding YAML pipeline file:

name: build-and-test
trigger: push
steps:
  - name: lint
    command: golangci-lint run ./...
    timeout: 2m
  - name: test
    command: go test -v -cover ./...
    timeout: 5m
  - name: build
    command: go build -o dist/app ./cmd/server
    timeout: 3m
artifacts:
  - dist/app

Step Execution and Isolation

Each step runs as an isolated subprocess using os/exec. A failing step doesn't crash the runner. The runner captures the exit code, records it, and decides whether to continue or stop. That's exactly what CI systems do.

package pipeline

import (
    "bytes"
    "context"
    "fmt"
    "os"
    "os/exec"
    "syscall"
    "time"
)

type StepResult struct {
    Name     string
    ExitCode int
    Output   string
    Duration time.Duration
    Err      error
}

func ExecuteStep(ctx context.Context, step Step, workDir string) StepResult {
    start := time.Now()

    timeout := 5 * time.Minute
    if step.Timeout != "" {
        if d, err := time.ParseDuration(step.Timeout); err == nil {
            timeout = d
        }
    }

    ctx, cancel := context.WithTimeout(ctx, timeout)
    defer cancel()

    cmd := exec.CommandContext(ctx, "sh", "-c", step.Command)
    cmd.Dir = workDir
    cmd.Env = append(os.Environ(), step.Env...)
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} // set process group

    var output bytes.Buffer
    cmd.Stdout = &output
    cmd.Stderr = &output

    err := cmd.Run()

    result := StepResult{
        Name:     step.Name,
        Output:   output.String(),
        Duration: time.Since(start),
    }

    if err != nil {
        if ctx.Err() != nil {
            // Context was cancelled or timed out: kill the entire process group
            killGroup(cmd)
        }
        result.Err = err
        if exitErr, ok := err.(*exec.ExitError); ok {
            result.ExitCode = exitErr.ExitCode()
        } else {
            result.ExitCode = -1
        }
    }

    fmt.Printf("[%s] exit=%d duration=%v\n", result.Name, result.ExitCode, result.Duration)
    // Output: [lint] exit=0 duration=1.23s
    return result
}

The context.WithTimeout ensures runaway commands are killed. The shell wrapper (sh -c) lets pipeline authors use pipes and redirects in their commands.

One gotcha: when the context is cancelled, Go sends SIGKILL to the process started by exec.CommandContext, but that process is sh -c. The child process it spawned, your actual command, may not receive the signal and keeps running. That's why SysProcAttr with Setpgid: true is there. It puts sh and all its children into the same process group. When you need to kill everything, you kill the whole group.

syscall is Go's low-level OS interface, used here to kill the entire subprocess group, not just the shell process. Note: SysProcAttr and syscall.Kill are Unix-only. If you need Windows support, guard this with a build constraint (//go:build !windows).

To kill the process group explicitly on cancellation, wrap the execution:

func killGroup(cmd *exec.Cmd) {
    if cmd.Process != nil {
        // Kill the entire process group with negative PID
        syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
    }
}

Call killGroup after cmd.Run() returns an error caused by context cancellation. This guarantees no orphaned subprocesses survive a timeout.

Artifact Management

After a successful build, artifacts need to be stored for downstream consumption. A simple artifact manager copies outputs to a structured directory, giving each run an isolated namespace so parallel runs don't collide:

package pipeline

import (
    "fmt"
    "io"
    "os"
    "path/filepath"
)

type ArtifactStore struct {
    BaseDir string
}

func (s *ArtifactStore) Store(runID string, patterns []string, workDir string) ([]string, error) {
    destDir := filepath.Join(s.BaseDir, runID)
    if err := os.MkdirAll(destDir, 0755); err != nil {
        return nil, fmt.Errorf("create artifact dir: %w", err)
    }

    var stored []string
    for _, pattern := range patterns {
        matches, err := filepath.Glob(filepath.Join(workDir, pattern))
        if err != nil {
            return nil, fmt.Errorf("glob %s: %w", pattern, err)
        }

        for _, src := range matches {
            dest := filepath.Join(destDir, filepath.Base(src))
            if err := copyFile(src, dest); err != nil {
                return nil, fmt.Errorf("copy %s: %w", src, err)
            }
            fmt.Printf("Stored artifact: %s\n", dest)
            // Output: Stored artifact: /tmp/ci-artifacts/run-42/app
            stored = append(stored, dest)
        }
    }
    return stored, nil
}

func copyFile(src, dst string) error {
    in, err := os.Open(src)
    if err != nil {
        return err
    }
    defer in.Close()

    out, err := os.Create(dst)
    if err != nil {
        return err
    }
    defer out.Close()

    _, err = io.Copy(out, in)
    return err
}

Status Reporting and Webhooks

CI systems report build status to external services (GitHub commit status, Slack, etc.) via webhooks. Decoupling the reporter from the runner means you can add new destinations without changing the execution logic:

package pipeline

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "time"
)

type BuildStatus struct {
    Pipeline   string       `json:"pipeline"`
    RunID      string       `json:"run_id"`
    Status     string       `json:"status"` // running, success, failed
    Steps      []StepResult `json:"steps"`
    DurationMs int64        `json:"duration_ms"`
}

type WebhookReporter struct {
    URL    string
    Client *http.Client
}

func (r *WebhookReporter) Report(status BuildStatus) error {
    body, err := json.Marshal(status)
    if err != nil {
        return fmt.Errorf("marshal status: %w", err)
    }

    req, err := http.NewRequest(http.MethodPost, r.URL, bytes.NewReader(body))
    if err != nil {
        return fmt.Errorf("create request: %w", err)
    }
    req.Header.Set("Content-Type", "application/json")

    resp, err := r.Client.Do(req)
    if err != nil {
        return fmt.Errorf("send webhook: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode >= 400 {
        return fmt.Errorf("webhook returned %d", resp.StatusCode)
    }
    fmt.Printf("Webhook sent: %s (status %d)\n", r.URL, resp.StatusCode)
    // Output: Webhook sent: https://hooks.slack.com/builds (status 200)
    return nil
}

Putting It Together: the Local Runner

The version you push to CI and the version you run locally are the same binary. Wire it as the devctl pipeline run subcommand:

func newPipelineRunCmd() *cobra.Command {
    cmd := &cobra.Command{
        Use:   "run <pipeline.yaml>",
        Short: "Run a CI/CD pipeline locally",
        Args:  cobra.ExactArgs(1),
        RunE: func(cmd *cobra.Command, args []string) error {
            return runPipeline(cmd.Context(), args[0], ".")
        },
    }
    return cmd
}

var pipelineCmd = &cobra.Command{Use: "pipeline", Short: "Pipeline commands"}

func init() {
    pipelineCmd.AddCommand(newPipelineRunCmd())
    rootCmd.AddCommand(pipelineCmd)
}

To debug a failing lint step:

./runner pipeline.yaml ./my-project --step lint

No commit. No push. No four-minute wait. You see the output immediately, fix the code, and run it again.

That's the payoff of building the runner in Go instead of relying on GitHub Actions YAML. The execution logic is testable, portable, and yours.

package main

import (
    "context"
    "fmt"
    "net/http"
    "os"
    "time"

    "gopkg.in/yaml.v3"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "usage: runner <pipeline.yaml> [work-dir]")
        os.Exit(1)
    }
    pipelineFile := os.Args[1]
    workDir := "."
    if len(os.Args) >= 3 {
        workDir = os.Args[2]
    }

    data, err := os.ReadFile(pipelineFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "read pipeline: %v\n", err)
        os.Exit(1)
    }

    var p Pipeline
    if err := yaml.Unmarshal(data, &p); err != nil {
        fmt.Fprintf(os.Stderr, "parse pipeline: %v\n", err)
        os.Exit(1)
    }

    ctx := context.Background()
    store := &ArtifactStore{BaseDir: "/tmp/ci-artifacts"}
    runID := fmt.Sprintf("run-%d", time.Now().UnixMilli())

    var results []StepResult
    failed := false
    for _, step := range p.Steps {
        result := ExecuteStep(ctx, step, workDir)
        results = append(results, result)
        if result.ExitCode != 0 {
            fmt.Fprintf(os.Stderr, "step %q failed (exit %d)\n", step.Name, result.ExitCode)
            failed = true
            break
        }
    }

    if !failed && len(p.Artifacts) > 0 {
        if _, err := store.Store(runID, p.Artifacts, workDir); err != nil {
            fmt.Fprintf(os.Stderr, "store artifacts: %v\n", err)
        }
    }

    if webhookURL := os.Getenv("WEBHOOK_URL"); webhookURL != "" {
        status := "success"
        if failed {
            status = "failed"
        }
        reporter := &WebhookReporter{URL: webhookURL, Client: &http.Client{Timeout: 10 * time.Second}}
        if err := reporter.Report(BuildStatus{Pipeline: p.Name, RunID: runID, Status: status, Steps: results}); err != nil {
            fmt.Fprintf(os.Stderr, "webhook: %v\n", err)
        }
    }

    if failed {
        os.Exit(1)
    }
}

YAML Pipeline Runner

A fully functional CI runner that loads a YAML pipeline definition, executes steps sequentially with timeout enforcement, stores build artifacts on success, and reports status via webhooks. It stops on the first failing step and exits with a non-zero code for CI integration.

Input: A YAML pipeline file path and an optional working directory (e.g., runner pipeline.yaml ./my-project). Optionally set WEBHOOK_URL env var for status reporting.

Output: Executes each step as a subprocess, logs exit codes and durations, stores artifacts to /tmp/ci-artifacts/<run-id>/, sends a JSON webhook on completion, and exits 0 on success or 1 on failure.

Full source: examples/yaml-pipeline-runner

Key Takeaways

CI runners are execution loops: parse definition, run steps, collect artifacts, report status. Build yours as a Go binary and you can run it locally.
Use os/exec with context timeouts for safe subprocess isolation.
When the context is cancelled, exec.CommandContext kills the top-level process but not its children. Set SysProcAttr{Setpgid: true} and kill the process group with syscall.Kill(-pid, syscall.SIGKILL) to avoid orphaned subprocesses.
Model pipelines as Go structs with YAML tags for easy serialization.
Artifact management is structured file copying with per-run namespacing to prevent collision.
Webhook reporters decouple the runner from notification concerns.
For programmatic pipelines, a builder DSL (NewPipeline("build").AddStep("test", "go test ./...").WithArtifacts("dist/app")) is a clean alternative to YAML files.
The entire runner pattern fits in under 200 lines. Production systems add caching, parallelism, and container isolation on top.

🎁 devctl pipeline run executes your pipeline locally. Next: devctl rollout deploy <service> <version> — ship the binary that pipeline just built, with automatic rollback if anything goes wrong.