06 - Building a K8s Operator

📋 Jump to Takeaways

🎁 What if you could turn your team's deployment runbook into a program that runs itself, forever, without anyone on-call?

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Operator Use Cases

Operators encode operational knowledge as code. Instead of documenting runbooks for scaling, upgrading, backing up, or recovering a service, you write a controller that performs those actions automatically. The operator watches your custom resources and continuously reconciles the cluster state to match the desired state you declared.

Examples of operator responsibilities: provisioning databases when a Database CR appears, rotating TLS certificates before expiry, scaling workers based on queue depth, or orchestrating blue-green deployments. Anything a human SRE does repeatedly becomes a reconciliation loop.

Custom Resource Definitions

A CRD extends the Kubernetes API with your own resource type. Here's a CRD for a simple web application:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: webapps.apps.example.com
spec:
  group: apps.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                port:
                  type: integer
            status:
              type: object
              properties:
                readyReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: webapps
    singular: webapp
    kind: WebApp
    shortNames:
      - wa

Apply it: kubectl apply -f webapp-crd.yaml. Now users can create WebApp resources.

Reconciliation Loop Pattern

Every operator follows the same pattern:

Watch: observe changes to your CR (and owned resources)
Compare: diff desired state (CR spec) against actual state (cluster resources)
Act: create, update, or delete resources to converge
Report: update the CR's status subresource

The reconciler is called whenever something changes. It must be idempotent. Calling it multiple times with the same input produces the same result. Never assume order or count of invocations.

Controller-Runtime Framework

controller-runtime (part of the Kubebuilder ecosystem) provides the scaffolding for building operators without boilerplate:

go get sigs.k8s.io/controller-runtime@latest

Key components:

Component	Role
Manager	Runs controllers, manages shared caches, handles leader election
Controller	Watches resources, queues reconcile requests
Reconciler	Your logic, called with a resource name/namespace
Client	Read/write K8s resources with caching

Go Types for the Custom Resource

Define Go structs matching your CRD schema:

package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// WebApp is the Schema for the webapps API
type WebApp struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   WebAppSpec   `json:"spec,omitempty"`
	Status WebAppStatus `json:"status,omitempty"`
}

type WebAppSpec struct {
	Image    string `json:"image"`
	Replicas int32  `json:"replicas"`
	Port     int32  `json:"port"`
}

type WebAppStatus struct {
	ReadyReplicas int32            `json:"readyReplicas"`
	Conditions    []metav1.Condition `json:"conditions,omitempty"`
}

// WebAppList contains a list of WebApp
type WebAppList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []WebApp `json:"items"`
}

import "k8s.io/apimachinery/pkg/runtime"

var SchemeBuilder = runtime.NewSchemeBuilder(func(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(
		schema.GroupVersion{Group: "apps.example.com", Version: "v1"},
		&WebApp{},
		&WebAppList{},
	)
	return nil
})

Reconciler Implementation

The reconciler is where your operational logic lives:

package controller

import (
	"context"
	"fmt"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/client-go/tools/record"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	webappv1 "example.com/operator/api/v1"
)

type WebAppReconciler struct {
	client.Client
	Recorder record.EventRecorder
}

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the WebApp CR
	var webapp webappv1.WebApp
	if err := r.Get(ctx, req.NamespacedName, &webapp); err != nil {
		if errors.IsNotFound(err) {
			// CR deleted, owned resources get garbage collected via OwnerReference
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	// 2. Build the desired Deployment
	desired := r.desiredDeployment(&webapp)

	// 3. Check if Deployment exists
	var existing appsv1.Deployment
	err := r.Get(ctx, types.NamespacedName{
		Name:      desired.Name,
		Namespace: desired.Namespace,
	}, &existing)

	if errors.IsNotFound(err) {
		// Create
		logger.Info("Creating Deployment", "name", desired.Name)
		// Output: INFO Creating Deployment {"name": "my-service-deployment"}
		if err := r.Create(ctx, desired); err != nil {
			return ctrl.Result{}, err
		}
	} else if err != nil {
		return ctrl.Result{}, err
	} else {
		// Update if spec changed
		if *existing.Spec.Replicas != webapp.Spec.Replicas ||
			existing.Spec.Template.Spec.Containers[0].Image != webapp.Spec.Image {
			existing.Spec.Replicas = &webapp.Spec.Replicas
			existing.Spec.Template.Spec.Containers[0].Image = webapp.Spec.Image
			logger.Info("Updating Deployment", "name", existing.Name)
			// Output: INFO Updating Deployment {"name": "my-service-deployment"}
			if err := r.Update(ctx, &existing); err != nil {
				return ctrl.Result{}, err
			}
		}
	}

	// 4. Update status
	webapp.Status.ReadyReplicas = existing.Status.ReadyReplicas
	if err := r.Status().Update(ctx, &webapp); err != nil {
		logger.Error(err, "updating status")
	}

	return ctrl.Result{}, nil
}

func (r *WebAppReconciler) desiredDeployment(webapp *webappv1.WebApp) *appsv1.Deployment {
	labels := map[string]string{
		"app":        webapp.Name,
		"managed-by": "webapp-operator",
	}

	dep := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      fmt.Sprintf("%s-deployment", webapp.Name),
			Namespace: webapp.Namespace,
			Labels:    labels,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &webapp.Spec.Replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: labels,
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name:  "app",
							Image: webapp.Spec.Image,
							Ports: []corev1.ContainerPort{
								{ContainerPort: webapp.Spec.Port},
							},
						},
					},
				},
			},
		},
	}

	// Set owner reference for garbage collection
	ctrl.SetControllerReference(webapp, dep, r.Scheme())
	return dep
}

// SetupWithManager registers the controller
func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.WebApp{}).        // Watch WebApp CRs
		Owns(&appsv1.Deployment{}).      // Watch owned Deployments
		Complete(r)
}

Status Updates and Events

Status communicates the actual state back to the user. Update it on every reconcile:

import "k8s.io/apimachinery/pkg/api/meta"

// Set a condition on the status
meta.SetStatusCondition(&webapp.Status.Conditions, metav1.Condition{
	Type:               "Available",
	Status:             metav1.ConditionTrue,
	Reason:             "DeploymentReady",
	Message:            "All replicas are available",
	LastTransitionTime: metav1.Now(),
})

r.Status().Update(ctx, &webapp)

Emit events for visibility in kubectl describe. The Recorder field we defined earlier on WebAppReconciler is a record.EventRecorder. Use it in your reconcile loop:

// In Reconcile(), after a successful create or update:
r.Recorder.Event(&webapp, corev1.EventTypeNormal, "Reconciled",
	fmt.Sprintf("Deployment %s updated to %d replicas", dep.Name, webapp.Spec.Replicas))
// Visible in: kubectl describe webapp my-service
// Events:
//   Normal  Reconciled  Deployment my-service-deployment updated to 5 replicas

Testing Operators with envtest

envtest spins up a real API server and etcd for integration testing without a full cluster:

package controller_test

import (
	"context"
	"testing"
	"time"

	. "github.com/onsi/gomega"
	appsv1 "k8s.io/api/apps/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/envtest"

	webappv1 "example.com/operator/api/v1"
	"example.com/operator/internal/controller"
)

func TestWebAppReconciler(t *testing.T) {
	g := NewWithT(t)

	// Start envtest environment
	testEnv := &envtest.Environment{
		CRDDirectoryPaths: []string{"../config/crd"},
	}

	cfg, err := testEnv.Start()
	g.Expect(err).NotTo(HaveOccurred())
	defer testEnv.Stop()

	// Create a client
	k8sClient, err := client.New(cfg, client.Options{})
	g.Expect(err).NotTo(HaveOccurred())

	// Start the manager and controller (in background)
	mgr, err := ctrl.NewManager(cfg, ctrl.Options{})
	g.Expect(err).NotTo(HaveOccurred())

	err = (&controller.WebAppReconciler{
		Client:   mgr.GetClient(),
		Recorder: mgr.GetEventRecorderFor("webapp-operator"),
	}).SetupWithManager(mgr)
	g.Expect(err).NotTo(HaveOccurred())

	go func() { mgr.Start(context.Background()) }()

	// Create a WebApp CR
	webapp := &webappv1.WebApp{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "test-app",
			Namespace: "default",
		},
		Spec: webappv1.WebAppSpec{
			Image:    "nginx:1.25",
			Replicas: 3,
			Port:     80,
		},
	}

	err = k8sClient.Create(context.Background(), webapp)
	g.Expect(err).NotTo(HaveOccurred())

	// Verify the Deployment was created
	deployment := &appsv1.Deployment{}
	g.Eventually(func() error {
		return k8sClient.Get(context.Background(), types.NamespacedName{
			Name:      "test-app-deployment",
			Namespace: "default",
		}, deployment)
	}, 10*time.Second, 250*time.Millisecond).Should(Succeed())

	g.Expect(*deployment.Spec.Replicas).To(Equal(int32(3)))
	g.Expect(deployment.Spec.Template.Spec.Containers[0].Image).To(Equal("nginx:1.25"))
}

Main Entrypoint

Wire everything together in main.go:

package main

import (
	"os"

	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"

	webappv1 "example.com/operator/api/v1"
	"example.com/operator/internal/controller"
)

func main() {
	ctrl.SetLogger(zap.New())
	logger := ctrl.Log.WithName("setup")

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		LeaderElection:   true,
		LeaderElectionID: "webapp-operator-lock",
	})
	if err != nil {
		logger.Error(err, "unable to create manager")
		os.Exit(1)
	}

	// Register CRD types
	webappv1.SchemeBuilder.AddToScheme(mgr.GetScheme())

	// Setup reconciler
	if err := (&controller.WebAppReconciler{
		Client:   mgr.GetClient(),
		Recorder: mgr.GetEventRecorderFor("webapp-operator"),
	}).SetupWithManager(mgr); err != nil {
		logger.Error(err, "unable to create controller")
		os.Exit(1)
	}

	logger.Info("starting manager")
	// Output: INFO starting manager
	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		logger.Error(err, "manager exited with error")
		os.Exit(1)
	}
}

Deploy a WebApp resource and the operator creates and manages the underlying Deployment:

apiVersion: apps.example.com/v1
kind: WebApp
metadata:
  name: my-service
spec:
  image: myregistry/myapp:v2.1.0
  replicas: 5
  port: 8080

Apply the resource and verify the operator created the Deployment:

kubectl apply -f webapp.yaml
kubectl get webapps
kubectl get deployments  # my-service-deployment with 5 replicas

WebApp Operator

The reconciler and main entrypoint together form a complete operator that watches WebApp custom resources and manages Kubernetes Deployments. It handles creation, updates, status reporting, and garbage collection via owner references.

Input: A WebApp CR specifying image, replicas, and port. Output: A managed Deployment that converges to match the CR spec.

Full source: examples/webapp-operator

Key Takeaways

Operators encode operational knowledge as code, turning runbooks into automated reconciliation loops
The reconciliation pattern is simple: fetch desired state, compare with actual state, converge, update status
controller-runtime handles the boilerplate (caching, leader election, event queuing) so you focus on business logic
Owner references enable automatic garbage collection. Delete the CR and all child resources clean up
Always update the status subresource and emit events so users can observe operator behavior via kubectl
envtest provides real API server testing without a cluster. Use it to verify your reconciler logic in CI
The For() and Owns() builder pattern automatically triggers reconciliation when either the CR or its owned resources change

🎁 What if you could describe your entire cloud infrastructure in Go, diff it against reality, and apply only the changes needed?