04 - Container Image Building

🎁 What if your deployment tool could build and push a container image as a single library call — no Docker installed, no daemon running, no Dockerfile?

The Docker SDK from the previous lesson powers devctl container deploy. This lesson adds devctl image build <binary> <registry-ref> — a subcommand that takes a compiled binary, packages it into an OCI image, and pushes it to a registry, all from Go code with no external tools.

Why would you want that? Three scenarios come up constantly in platform engineering:

CI running inside containers. Your pipeline runs jobs in containers, and you add a docker build step. It fails with Cannot connect to the Docker daemon. Mounting the host socket is a security risk most teams block. Docker-in-Docker (the classic Jenkins approach — running a full Docker daemon inside the CI container) requires --privileged mode, causes storage driver conflicts with the outer runtime, and is why Jenkins pipelines were notoriously fragile. Modern CI systems (GitHub Actions, GitLab CI, Google Cloud Build) moved away from it. Tools like Kaniko and Buildah, and go-containerregistry under the hood, all work the same way: write the OCI format directly, push over HTTPS, no daemon needed.

Deployment tools that also ship images. Your devctl release command compiles the binary, builds the image, pushes it, and updates the Kubernetes deployment — all in one command, no Docker required on the machine running it.

Minimal production images. A tool that builds an image programmatically can produce a scratch-based image containing only your binary — no shell, no OS, no package manager. The resulting image is 5-15 MB with zero attack surface. docker build with a Dockerfile can do this too, but it requires Docker. go-containerregistry requires nothing.

The underlying format makes this possible. An OCI image is a manifest, a config blob, and a set of compressed tar archives. It's JSON and bytes over HTTPS. Any program that can write tar files and make HTTP requests can build and push an image.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

The Docker-in-Docker Problem

Your CI pipeline needs to build a container image. The pipeline itself runs inside a container. You have three options, and two of them are bad.

Option 1: Mount the host Docker socket. Works, but any process in your CI container can now talk to the host daemon — create privileged containers, mount host paths, escape isolation. Most security teams block this.

Option 2: Docker-in-Docker. Run a full Docker daemon inside the CI container. This is the classic Jenkins approach: the docker:dind sidecar, --privileged mode, storage driver conflicts between the inner and outer runtimes. It's why Jenkins image builds were fragile and slow. GitHub Actions, GitLab CI, and Google Cloud Build all moved away from it.

Option 3: Build without a daemon. This is what Kaniko, Buildah, and go-containerregistry do. An OCI image is just a manifest, a config blob, and compressed tar layers — it's JSON and bytes over HTTPS. You don't need a daemon to assemble and push that. You need a library that speaks the registry protocol.

OCI (Open Container Initiative) is the industry standard that all registries speak — Docker Hub, GCR, ECR, Artifactory, GHCR. go-containerregistry is a pure-Go implementation of that protocol. Your CI job, your deployment tool, or your release script can build and push images to any of these registries with no Docker installed anywhere.

go get github.com/google/go-containerregistry

Key packages:

Creating Layers Programmatically

A layer is a gzip-compressed tar archive representing a filesystem snapshot. You create one by writing file entries into a tar writer and handing the result to the library:

func createLayer(files map[string][]byte) (v1.Layer, error) {
	var buf bytes.Buffer
	gw := gzip.NewWriter(&buf)
	tw := tar.NewWriter(gw)

	keys := make([]string, 0, len(files))
	for name := range files {
		keys = append(keys, name)
	}
	sort.Strings(keys)

	for _, name := range keys {
		content := files[name]
		header := &tar.Header{
			Name:    name,
			Size:    int64(len(content)),
			Mode:    0755,
			ModTime: time.Now(),
		}
		if err := tw.WriteHeader(header); err != nil {
			return nil, err
		}
		if _, err := tw.Write(content); err != nil {
			return nil, err
		}
	}

	tw.Close()
	gw.Close()

	return tarball.LayerFromReader(&buf)
}

Gotcha: The sort.Strings(keys) before writing is not just tidiness. If the same set of files is written in different order, the tar archive has different bytes, so the SHA-256 digest changes, and the layer gets a new hash even though the content is identical. That defeats layer caching at the registry. The push succeeds but uploads redundant data on every build. Always write files in deterministic order.

Mutating Base Images

Starting from a raw layer works for a scratch image that contains only your binary. For anything that needs a base OS filesystem, you pull a real base image and append your layer on top.

func buildImage(binaryLayer v1.Layer) (v1.Image, error) {
	// Start from scratch (empty image)
	base := empty.Image

	img, err := mutate.AppendLayers(base, binaryLayer)
	if err != nil {
		return nil, err
	}

	cfg, err := img.ConfigFile()
	if err != nil {
		return nil, err
	}

	cfg.Config.Entrypoint = []string{"/app/server"}
	cfg.Config.Env = []string{
		"PORT=8080",
		"ENV=production",
	}
	cfg.Config.WorkingDir = "/app"
	cfg.Config.ExposedPorts = map[string]struct{}{
		"8080/tcp": {},
	}

	return mutate.ConfigFile(img, cfg)
}

To use a real base image instead of scratch:

ref, _ := name.ParseReference("gcr.io/distroless/static:nonroot")
base, _ := remote.Image(ref)
img, _ := mutate.AppendLayers(base, myLayer)

Gotcha: mutate.ConfigFile replaces the entire config, not just the fields you changed. If you construct a new v1.ConfigFile{} from scratch and pass it in, you lose the platform list (linux/amd64, linux/arm64) and any OS-level fields the base image carried. Always read the existing config with img.ConfigFile() first, mutate only the fields you need, then write it back.

Pushing to Registries with Authentication

The remote package handles the full push sequence: uploading each layer blob, uploading the config blob, then writing the manifest. You hand it the image and a destination reference and it handles the rest.

func pushImage(img v1.Image, destination string) error {
	ref, err := name.ParseReference(destination)
	if err != nil {
		return err
	}

	// Uses ~/.docker/config.json or credential helpers
	return remote.Write(ref, img,
		remote.WithAuthFromKeychain(authn.DefaultKeychain),
	)
}

authn.DefaultKeychain reads from ~/.docker/config.json and invokes any credential helpers configured there, the same lookup chain docker push uses. For CI environments with no local Docker config, pass credentials directly:

auth := &authn.Basic{
	Username: "oauth2accesstoken",
	Password: os.Getenv("REGISTRY_TOKEN"),
}

err := remote.Write(ref, img, remote.WithAuth(auth))

Gotcha: If you also want to tag the image as latest, don't call remote.Write twice. A second write re-uploads all layer blobs, wasting bandwidth, even though the registry deduplicates them. Use remote.Tag(latestRef, img) instead. It writes only the manifest, pointing to the blobs that are already there.

Putting It Together: devctl image build

Here's what your CI build step becomes: one function that reads a compiled binary, builds a scratch image, and pushes it to a registry — wired into devctl as the image build subcommand. No Dockerfile, no Docker socket, no privileged container required.

func buildAndPush(binaryPath, destination string) error {
	data, err := os.ReadFile(binaryPath)
	if err != nil {
		return fmt.Errorf("read binary: %w", err)
	}

	layer, err := createLayer(map[string][]byte{
		"app/server": data,
	})
	if err != nil {
		return fmt.Errorf("create layer: %w", err)
	}

	img, err := buildImage(layer)
	if err != nil {
		return fmt.Errorf("build image: %w", err)
	}

	digest, err := img.Digest()
	if err != nil {
		return fmt.Errorf("digest: %w", err)
	}

	fmt.Printf("pushing %s (digest: %s)...\n", destination, digest)

	if err := pushImage(img, destination); err != nil {
		return fmt.Errorf("push: %w", err)
	}

	fmt.Printf("pushed successfully\n")
	return nil
}

The digest is printed before the push. If the push fails and you retry, the same binary produces the same layer digest, so the registry deduplicates the blobs. You can retry safely without creating a different image.

Wire buildAndPush into devctl as the image build subcommand:

var imageCmd = &cobra.Command{
    Use:   "image",
    Short: "Build and push container images",
}

var imageBuildCmd = &cobra.Command{
    Use:   "build <binary-path> <registry-ref>",
    Short: "Build a scratch OCI image from a static binary and push it",
    Args:  cobra.ExactArgs(2),
    RunE: func(cmd *cobra.Command, args []string) error {
        return buildAndPush(args[0], args[1])
    },
}

func init() {
    imageCmd.AddCommand(imageBuildCmd)
    rootCmd.AddCommand(imageCmd)
}

Now devctl image build ./myserver ghcr.io/myorg/myapp:v1.0.0 reads the binary, builds the OCI image, and pushes it — auth comes from ~/.docker/config.json or the REGISTRY_TOKEN env var, same lookup chain as docker push.

To try it locally, push to a local Docker registry. Start one with:

docker run -d -p 5000:5000 --name registry registry:2

Build a static binary and push the image to it:

# build a static binary (CGO_ENABLED=0 required for scratch images)
CGO_ENABLED=0 go build -o myserver .

# build the OCI image and push to your local registry
go run main.go ./myserver localhost:5000/myapp:latest

# verify it arrived
docker pull localhost:5000/myapp:latest
docker run --rm localhost:5000/myapp:latest

Expected output:

pushing localhost:5000/myapp:latest (digest: sha256:abc123...)...
pushed successfully

Once pushed, you can inspect what's in your local registry using the OCI Distribution API directly — no Docker required:

# list all repositories in the registry
curl http://localhost:5000/v2/_catalog
# {"repositories":["myapp"]}

# list all tags for your image
curl http://localhost:5000/v2/myapp/tags/list
# {"name":"myapp","tags":["latest"]}

# fetch the image manifest (layers, config digest, media type)
curl http://localhost:5000/v2/myapp/manifests/latest \
  -H "Accept: application/vnd.oci.image.manifest.v1+json"

These are the same HTTP endpoints go-containerregistry calls when it pushes. The registry is just a REST API — _catalog, tags/list, and manifests/:tag are the three endpoints you'll use most when debugging.

If you prefer Docker commands:

docker pull localhost:5000/myapp:latest
docker image inspect localhost:5000/myapp:latest
docker run --rm localhost:5000/myapp:latest

Scratch Image Builder

A program that builds minimal OCI container images entirely in Go, no Docker daemon required. It reads a pre-compiled static binary, constructs a tar layer, assembles a scratch-based image with entrypoint and environment config, and pushes the result to a container registry. The output is a 5-15 MB image with zero OS dependencies.

Input: A statically-compiled Go binary path and a destination registry reference (e.g., go run main.go myserver ghcr.io/myorg/myapp:v1.0.0)

Output: A pushed OCI image with digest confirmation, containing only your binary in a scratch container

Full source: examples/scratch-image-builder

Key Takeaways

• OCI images are manifests + config + ordered tar.gz layers, a format you can produce entirely in Go without a daemon
• go-containerregistry handles the distribution spec: chunked uploads, content digests, auth token challenges, and credential helper lookup
• Write tar file entries in sorted, deterministic order. Random order produces different digests and defeats layer caching at the registry
• Always read the existing config with img.ConfigFile() before mutating. Constructing a new config from scratch loses platform and OS fields from the base image
• To add a latest tag after pushing, use remote.Tag, not a second remote.Write. The second write wastes bandwidth re-uploading blobs that are already there
• A build pipeline that produces OCI images needs no Docker daemon, no privileged containers, and no socket mounts

🎁 devctl image build pushes your image to the registry. What if devctl k8s deploy could then tell Kubernetes to run it, watch the rollout, and report exactly what failed — without a kubectl binary anywhere in sight?