06 - Building a K8s Operator

📋 Jump to Takeaways

🎁 Your team has a 12-step runbook for deploying a database cluster. At 3am during an incident, someone will skip step 7.

The problem isn't documentation. The problem is that humans follow runbooks inconsistently — under pressure, out of order, on the wrong version. As your platform grows, runbooks multiply and diverge. Each one is a liability waiting to become an incident.

An operator replaces the runbook with code. The "steps" become a reconciliation loop that runs continuously, catches drift, and brings your cluster back to the declared state. No one has to follow it correctly. It runs itself.

Your team is building devctl, a CLI that packages all platform operations into one tool. The operator runs as a long-lived daemon pod, so devctl gets an operator install subcommand that deploys it to the cluster once.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Why Not a Helm Chart or CronJob?

If you've used Helm, you've solved part of the problem. Helm packages Kubernetes resources and templates values. But Helm deploys and walks away. If someone manually scales your Deployment to zero, Helm has no idea. It doesn't watch.

CronJobs run on a schedule. But a database needs its credentials rotated when they expire, not at midnight regardless of what else is happening in the cluster. A backup should happen before an upgrade, not on a fixed timer.

Operators solve both problems:

Event-driven: the reconciler runs whenever something changes, not on a schedule
Self-healing: if someone manually deletes a resource your operator owns, the operator recreates it
Operationally aware: you can encode logic like "back up before upgrading" or "don't scale during a migration"

The rule of thumb: if your runbook has conditional steps, that logic belongs in an operator.

Custom Resource Definitions

A CRD extends the Kubernetes API with your own resource type. Once applied, users can kubectl apply a WebApp the same way they'd apply a Deployment.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: webapps.apps.example.com
spec:
  group: apps.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                port:
                  type: integer
            status:
              type: object
              properties:
                readyReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: webapps
    singular: webapp
    kind: WebApp
    shortNames:
      - wa

Apply it: kubectl apply -f webapp-crd.yaml. Now users can create WebApp resources.

Controller-Runtime Setup

controller-runtime is the framework most Kubernetes operators use. It handles the watch machinery, caching, leader election, and request queuing. You implement one method: Reconcile.

go get sigs.k8s.io/controller-runtime@latest

The four components you work with:

Component	Role
Manager	Runs controllers, manages shared caches, handles leader election
Controller	Watches resources, queues reconcile requests
Reconciler	Your logic, called with a resource name/namespace
Client	Read/write K8s resources with caching

The Reconciliation Loop and Idempotency

Every operator follows the same pattern:

Watch: observe changes to your CR and its owned resources
Compare: diff desired state (CR spec) against actual state (cluster resources)
Act: create, update, or delete resources to converge
Report: update the CR's status subresource

The reconciler is called whenever something changes. The critical rule: it must be idempotent. The same reconcile call with the same input must always produce the same result.

This is where people get burned. You can't assume the reconciler runs once for a Create event and once for an Update event. Kubernetes calls your reconciler multiple times for the same event, on restart, on leader election changes, and whenever any watched resource changes. Write it assuming it might be called five times in a row.

The wrong approach:

// ❌ BAD: assumes reconciler is called exactly once per create
func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    desired := r.desiredDeployment(&webapp)
    // This fails on the second call with "already exists"
    if err := r.Create(ctx, desired); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

The correct approach checks first, then branches:

// ✅ GOOD: get first, then create or update
var existing appsv1.Deployment
err := r.Get(ctx, types.NamespacedName{
    Name:      desired.Name,
    Namespace: desired.Namespace,
}, &existing)

if errors.IsNotFound(err) {
    // First time: create it
    return ctrl.Result{}, r.Create(ctx, desired)
} else if err != nil {
    return ctrl.Result{}, err
} else {
    // Already exists: update only if spec changed
    if *existing.Spec.Replicas != webapp.Spec.Replicas ||
        existing.Spec.Template.Spec.Containers[0].Image != webapp.Spec.Image {
        existing.Spec.Replicas = &webapp.Spec.Replicas
        existing.Spec.Template.Spec.Containers[0].Image = webapp.Spec.Image
        return ctrl.Result{}, r.Update(ctx, &existing)
    }
}

Every resource your operator manages needs this pattern. Every resource, every time.

Go Types for Your Custom Resource

Your Go structs mirror the CRD schema. The controller-runtime client uses these for serialization.

package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type WebApp struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   WebAppSpec   `json:"spec,omitempty"`
	Status WebAppStatus `json:"status,omitempty"`
}

type WebAppSpec struct {
	Image    string `json:"image"`
	Replicas int32  `json:"replicas"`
	Port     int32  `json:"port"`
}

type WebAppStatus struct {
	ReadyReplicas int32              `json:"readyReplicas"`
	Conditions    []metav1.Condition `json:"conditions,omitempty"`
}

type WebAppList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []WebApp `json:"items"`
}

var SchemeBuilder = runtime.NewSchemeBuilder(func(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(
		schema.GroupVersion{Group: "apps.example.com", Version: "v1"},
		&WebApp{},
		&WebAppList{},
	)
	return nil
})

Reconciler Implementation

Here is the full reconciler. Notice the check-and-create-or-update pattern applied to both the Deployment and the Service:

type WebAppReconciler struct {
	client.Client
	Recorder record.EventRecorder
}

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// Fetch the WebApp CR
	var webapp webappv1.WebApp
	if err := r.Get(ctx, req.NamespacedName, &webapp); err != nil {
		if errors.IsNotFound(err) {
			// CR was deleted; owned resources are garbage collected via OwnerReference
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	// Reconcile Deployment
	desired := r.desiredDeployment(&webapp)
	var existing appsv1.Deployment
	err := r.Get(ctx, types.NamespacedName{
		Name: desired.Name, Namespace: desired.Namespace,
	}, &existing)

	if errors.IsNotFound(err) {
		logger.Info("Creating Deployment", "name", desired.Name)
		// Output: INFO Creating Deployment {"name": "my-service-deployment"}
		if err := r.Create(ctx, desired); err != nil {
			return ctrl.Result{}, err
		}
	} else if err != nil {
		return ctrl.Result{}, err
	} else {
		if *existing.Spec.Replicas != webapp.Spec.Replicas ||
			existing.Spec.Template.Spec.Containers[0].Image != webapp.Spec.Image {
			existing.Spec.Replicas = &webapp.Spec.Replicas
			existing.Spec.Template.Spec.Containers[0].Image = webapp.Spec.Image
			logger.Info("Updating Deployment", "name", existing.Name)
			// Output: INFO Updating Deployment {"name": "my-service-deployment"}
			if err := r.Update(ctx, &existing); err != nil {
				return ctrl.Result{}, err
			}
		}
	}

	// Reconcile Service (same pattern)
	desiredSvc := r.desiredService(&webapp)
	var existingSvc corev1.Service
	svcErr := r.Get(ctx, types.NamespacedName{
		Name: desiredSvc.Name, Namespace: desiredSvc.Namespace,
	}, &existingSvc)

	if errors.IsNotFound(svcErr) {
		logger.Info("Creating Service", "name", desiredSvc.Name)
		if err := r.Create(ctx, desiredSvc); err != nil {
			return ctrl.Result{}, err
		}
	} else if svcErr != nil {
		return ctrl.Result{}, svcErr
	}

	// Update status
	webapp.Status.ReadyReplicas = existing.Status.ReadyReplicas
	if err := r.Status().Update(ctx, &webapp); err != nil {
		logger.Error(err, "updating status")
	}

	return ctrl.Result{}, nil
}

The desiredDeployment helper builds the Deployment spec and calls ctrl.SetControllerReference, which sets the owner reference. When the WebApp CR is deleted, Kubernetes garbage-collects the Deployment and Service automatically. You don't write any cleanup code.

func (r *WebAppReconciler) desiredDeployment(webapp *webappv1.WebApp) *appsv1.Deployment {
	labels := map[string]string{
		"app":        webapp.Name,
		"managed-by": "webapp-operator",
	}
	dep := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      fmt.Sprintf("%s-deployment", webapp.Name),
			Namespace: webapp.Namespace,
			Labels:    labels,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &webapp.Spec.Replicas,
			Selector: &metav1.LabelSelector{MatchLabels: labels},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{
						Name:  "app",
						Image: webapp.Spec.Image,
						Ports: []corev1.ContainerPort{{ContainerPort: webapp.Spec.Port}},
					}},
				},
			},
		},
	}
	ctrl.SetControllerReference(webapp, dep, r.Scheme())
	return dep
}

The desiredService helper does the same for the Service: it selects pods by the webapp's name label and exposes port 80, targeting whatever port the webapp declares.

func (r *WebAppReconciler) desiredService(webapp *webappv1.WebApp) *corev1.Service {
	labels := map[string]string{
		"app":        webapp.Name,
		"managed-by": "webapp-operator",
	}
	svc := &corev1.Service{
		ObjectMeta: metav1.ObjectMeta{
			Name:      fmt.Sprintf("%s-svc", webapp.Name),
			Namespace: webapp.Namespace,
			Labels:    labels,
		},
		Spec: corev1.ServiceSpec{
			Selector: labels,
			Ports: []corev1.ServicePort{{
				Port:       80,
				TargetPort: intstr.FromInt32(webapp.Spec.Port),
			}},
		},
	}
	ctrl.SetControllerReference(webapp, svc, r.Scheme())
	return svc
}

Status, Events, and Observability

Users can't see inside your reconciler unless you tell them what it's doing. Two mechanisms exist: status conditions and events.

Status conditions live on the CR and reflect the current state. Set them on every reconcile so kubectl get webapp shows something meaningful:

import "k8s.io/apimachinery/pkg/api/meta"

meta.SetStatusCondition(&webapp.Status.Conditions, metav1.Condition{
	Type:               "Available",
	Status:             metav1.ConditionTrue,
	Reason:             "DeploymentReady",
	Message:            "All replicas are available",
	LastTransitionTime: metav1.Now(),
})

r.Status().Update(ctx, &webapp)

Events surface in kubectl describe and record individual actions:

r.Recorder.Event(&webapp, corev1.EventTypeNormal, "Reconciled",
	fmt.Sprintf("Deployment %s updated to %d replicas", dep.Name, webapp.Spec.Replicas))
// Visible in: kubectl describe webapp my-service
// Events:
//   Normal  Reconciled  Deployment my-service-deployment updated to 5 replicas

Write status conditions for the current state of the resource. Use events for individual actions ("created", "scaled", "backup started"). Both are cheap and the data is irreplaceable during an incident.

Testing with envtest

envtest runs a real Kubernetes API server and etcd in your test process. You get a cluster without needing one.

func TestWebAppReconciler(t *testing.T) {
	g := NewWithT(t)

	testEnv := &envtest.Environment{
		CRDDirectoryPaths: []string{"../config/crd"},
	}
	cfg, err := testEnv.Start()
	g.Expect(err).NotTo(HaveOccurred())
	defer testEnv.Stop()

	k8sClient, err := client.New(cfg, client.Options{})
	g.Expect(err).NotTo(HaveOccurred())

	mgr, err := ctrl.NewManager(cfg, ctrl.Options{})
	g.Expect(err).NotTo(HaveOccurred())

	err = (&controller.WebAppReconciler{
		Client:   mgr.GetClient(),
		Recorder: mgr.GetEventRecorderFor("webapp-operator"),
	}).SetupWithManager(mgr)
	g.Expect(err).NotTo(HaveOccurred())

	go func() { mgr.Start(context.Background()) }()

	webapp := &webappv1.WebApp{
		ObjectMeta: metav1.ObjectMeta{Name: "test-app", Namespace: "default"},
		Spec:       webappv1.WebAppSpec{Image: "nginx:1.25", Replicas: 3, Port: 80},
	}
	err = k8sClient.Create(context.Background(), webapp)
	g.Expect(err).NotTo(HaveOccurred())

	deployment := &appsv1.Deployment{}
	g.Eventually(func() error {
		return k8sClient.Get(context.Background(),
			types.NamespacedName{Name: "test-app-deployment", Namespace: "default"},
			deployment)
	}, 10*time.Second, 250*time.Millisecond).Should(Succeed())

	g.Expect(*deployment.Spec.Replicas).To(Equal(int32(3)))
	g.Expect(deployment.Spec.Template.Spec.Containers[0].Image).To(Equal("nginx:1.25"))
}

Also write an idempotency test: reconcile the same object twice and verify the second call doesn't create a duplicate or return an error. That test would have caught the BAD pattern shown earlier.

Wiring It Together

The main entrypoint creates the manager, registers your types, and starts the loop. Leader election is critical in production. Without it, multiple operator replicas reconcile the same resources simultaneously and step on each other.

In devctl, the operator daemon has its own binary, but devctl operator install deploys it to the cluster. Wire runOperator into Cobra like this:

var operatorCmd = &cobra.Command{
    Use:   "operator",
    Short: "Manage the WebApp operator",
}

var operatorInstallCmd = &cobra.Command{
    Use:   "install",
    Short: "Deploy the WebApp operator to the cluster",
    RunE: func(cmd *cobra.Command, args []string) error {
        return runOperator()
    },
}

func init() {
    operatorCmd.AddCommand(operatorInstallCmd)
    rootCmd.AddCommand(operatorCmd)
}

runOperator is the same function the daemon binary uses — install just triggers it once from the CLI:

func runOperator() error {
	ctrl.SetLogger(zap.New())
	logger := ctrl.Log.WithName("setup")

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		LeaderElection:   true,
		LeaderElectionID: "webapp-operator-lock",
	})
	if err != nil {
		logger.Error(err, "unable to create manager")
		return err
	}

	webappv1.SchemeBuilder.AddToScheme(mgr.GetScheme())

	if err := (&controller.WebAppReconciler{
		Client:   mgr.GetClient(),
		Recorder: mgr.GetEventRecorderFor("webapp-operator"),
	}).SetupWithManager(mgr); err != nil {
		logger.Error(err, "unable to create controller")
		return err
	}

	logger.Info("starting manager")
	// Output: INFO starting manager
	return mgr.Start(ctrl.SetupSignalHandler())
}

SetupWithManager registers the controller with the manager. The For and Owns calls tell it what to watch:

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&webappv1.WebApp{}).       // watch WebApp CRs
        Owns(&appsv1.Deployment{}).    // watch Deployments owned by a WebApp
        Owns(&corev1.Service{}).       // watch Services owned by a WebApp
        Complete(r)
}

Owning a Deployment means: if that Deployment is changed or deleted, queue a reconcile for the parent WebApp. If someone manually deletes your Deployment, the operator creates it again. The runbook is gone. There's code now.

Users interact only with the WebApp CR:

apiVersion: apps.example.com/v1
kind: WebApp
metadata:
  name: my-service
spec:
  image: myregistry/myapp:v2.1.0
  replicas: 5
  port: 8080

kubectl apply -f webapp.yaml
kubectl get webapps
kubectl get deployments  # my-service-deployment with 5 replicas

WebApp Operator

The reconciler and main entrypoint together form a complete operator that watches WebApp custom resources and manages Kubernetes Deployments. It handles creation, updates, status reporting, and garbage collection via owner references.

Input: A WebApp CR specifying image, replicas, and port. Output: A managed Deployment that converges to match the CR spec.

Full source: examples/webapp-operator

Key Takeaways

An operator replaces a runbook with code. The reconciliation loop runs continuously and self-heals, no human steps required.
Operators are event-driven and self-healing, unlike Helm charts (which deploy and walk away) or CronJobs (which are time-based, not change-based).
Reconcilers MUST be idempotent. Kubernetes calls yours multiple times for the same event. Use the get-then-create-or-update pattern for every resource you manage.
controller-runtime handles caching, leader election, and event queuing. You implement Reconcile.
Set owner references with ctrl.SetControllerReference so deleting the CR garbage-collects all owned resources. No cleanup code needed.
Always update status conditions and emit events. They are the user's only window into what the operator is doing.
Test with envtest, which runs a real API server in your test process. Include an idempotency test that reconciles the same object twice.

🎁 The operator handles runtime state in the cluster. What about the cloud infrastructure under it? Next, devctl infra provision uses the Pulumi Automation API to provision environments on demand, no console access required.