10 - Metrics & Prometheus

🎁 How do you answer "is the service slow?" with a number instead of a guess, and get alerted before users even notice?

The last subcommand being added to devctl is devctl metrics. It starts the Prometheus exporter so your platform tooling exposes queue depth, latency, and worker counts alongside the services it manages. You can also use devctl infra metrics to check queue depth before provisioning.

Your service is slow. Users are complaining. You SSH into the box and run top. CPU looks fine. You tail the logs. Nothing obvious. You check the database. Query times look normal.

An hour later you find it: the background job queue is 8000 items deep and has been growing for 90 minutes. The workers are running, but new jobs are arriving faster than they're being processed. Every user who triggered a job is waiting on a response that will never come quickly.

You had no metric for queue depth. Nobody told you the queue was a problem until it became a crisis. This is what happens without instrumentation: you find out about problems from users, not dashboards.

A queue depth gauge would have triggered an alert at 500 items. You'd have found this in 10 minutes, not 90.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Why Not Just expvar?

Go's standard library ships with expvar — short for "exported variables." It's a Go-specific package with no equivalent in other languages. It lets you expose internal counters and values from your running program at /debug/vars as JSON, with a single import and zero setup. You can open that URL in a browser and see live numbers from the process.

// expvar: useful for one-off debugging
import _ "expvar"

var queueDepth = expvar.NewInt("queue_depth")
queueDepth.Set(int64(len(jobQueue))) // visible at /debug/vars as JSON

That's useful for a quick sanity check on a single instance. But expvar doesn't support histograms, so you can't compute latency percentiles. It has no concept of label dimensions, so you can't slice by HTTP method or status code. And it outputs JSON in its own format, not the Prometheus text format that Prometheus scrapes.

Without Prometheus, there are no alerting rules. No rate() queries. No dashboards. expvar is a debugging aid, not an observability system.

The prometheus/client_golang library is what you need for production:

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

What Is Queue Depth?

Queue depth is the number of jobs waiting to be processed at a given moment. Think of it as the line of work that hasn't been picked up yet.

[job 1] [job 2] [job 3] ... [job 8000] → [worker]
         ↑ queue depth = 8000

If jobs arrive faster than your worker can process them, the depth grows. CPU and memory can look completely normal while this is happening — workers are busy, nothing is crashing. But users are waiting for responses that won't come for minutes. That's why the lesson opens with a 90-minute incident that could have been caught in 10: a single queue depth gauge with an alert threshold would have fired long before users noticed.

The pattern generalizes to anything that accumulates: pending emails, unprocessed webhooks, database write buffers, message broker lag. Wherever work can pile up, a gauge measuring the pile gives you early warning.

Prometheus Data Model

Prometheus organizes everything around time series. Each series is identified by a metric name and a set of key-value labels:

http_requests_total{method="GET", status="200"} 1234

Every unique combination of name and labels is a separate series. Labels are how you slice and dice: by method, status code, endpoint, instance. Queue depth becomes useful when it has a label for which queue, so you can ask "which queue is backing up?" instead of "is anything wrong?"

Prometheus scrapes your application's /metrics endpoint on a schedule, typically every 15-30 seconds. Your app is a passive target. It exposes numbers and waits.

Prometheus defines four metric types:

• Counter: monotonically increasing. Use for totals: requests served, errors, bytes sent. Never decreases except on restart.
• Gauge: goes up and down. Use for current state: queue depth, active connections, memory usage.
• Histogram: samples observations into configurable buckets. Use for latency and request sizes when you need percentiles.
• Summary: calculates quantiles client-side. Use when you need precise quantiles from a single instance.

Instrumenting Your Application

Define your metrics as package-level variables, register them in init(), then call them from your handlers. The registration step is what makes them visible at /metrics.

package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	requestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "myapp_requests_total",
			Help: "Total number of HTTP requests.",
		},
		[]string{"method", "path", "status"},
	)

	queueDepth = prometheus.NewGauge(
		prometheus.GaugeOpts{
			Name: "myapp_queue_depth",
			Help: "Number of jobs waiting in the queue.",
		},
	)

	requestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "myapp_request_duration_seconds",
			Help:    "Request latency in seconds.",
			Buckets: []float64{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0},
		},
		[]string{"method", "path"},
	)
)

func init() {
	prometheus.MustRegister(requestsTotal, queueDepth, requestDuration)
}

func main() {
	http.Handle("/metrics", promhttp.Handler())
	http.HandleFunc("/api/data", func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()

		// Simulate work and enqueue a background job
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		select {
		case jobQueue <- struct{}{}:
		default: // queue full — drop rather than block the handler
		}

		requestsTotal.WithLabelValues(r.Method, "/api/data", "200").Inc()
		requestDuration.WithLabelValues(r.Method, "/api/data").Observe(time.Since(start).Seconds())

		w.Write([]byte(`{"status":"ok"}`))
	})

	// jobQueue simulates a real work queue. The /api/data handler enqueues
	// a job on each request; a background worker drains it slowly so depth grows.
	jobQueue := make(chan struct{}, 1000)

	// Slow worker: drains one item every 200ms to simulate backlog buildup
	go func() {
		for range jobQueue {
			time.Sleep(200 * time.Millisecond)
		}
	}()

	// Update queue depth gauge every 5 seconds
	go func() {
		for {
			queueDepth.Set(float64(len(jobQueue)))
			time.Sleep(5 * time.Second)
		}
	}()

	fmt.Println("serving metrics on :8080/metrics")
	// Output: serving metrics on :8080/metrics
	http.ListenAndServe(":8080", nil)
}

Hit /metrics and you'll see:

# HELP myapp_queue_depth Number of jobs waiting in the queue.
# TYPE myapp_queue_depth gauge
myapp_queue_depth 8000
# HELP myapp_requests_total Total number of HTTP requests.
# TYPE myapp_requests_total counter
myapp_requests_total{method="GET",path="/api/data",status="200"} 42

That myapp_queue_depth 8000 would have triggered an alert. You would not have spent 90 minutes in the dark.

Gotcha: Don't use MustRegister in code that might run more than once. MustRegister panics if you try to register the same metric twice. In tests, use prometheus.NewRegistry() to get a clean slate and register metrics explicitly into it.

Label Cardinality

Labels are powerful but they have a hard limit: every unique label value creates a new time series. Prometheus stores and indexes each series separately. At scale, bad label design can take down your monitoring.

Here's the pattern that will OOM your Prometheus:

// ❌ WRONG: user_id as a label
requestsTotal = prometheus.NewCounterVec(
	prometheus.CounterOpts{Name: "myapp_requests_total"},
	[]string{"method", "path", "status", "user_id"}, // never do this
)

// 100,000 users x 3 methods x 10 paths x 3 status codes = 9,000,000 series
requestsTotal.WithLabelValues("GET", "/api/data", "200", "user-abc123").Inc()

The right approach: only use label values that come from a bounded set you control.

// ✅ CORRECT: only low-cardinality dimensions
requestsTotal = prometheus.NewCounterVec(
	prometheus.CounterOpts{Name: "myapp_requests_total"},
	[]string{"method", "path", "status"}, // bounded sets
)

// user_id belongs in logs, not metrics
// For per-user visibility, query your log system.
requestsTotal.WithLabelValues("GET", "/api/data", "200").Inc()

Good label candidates: HTTP methods, status codes, endpoint names, service names, regions. Bad label candidates: user IDs, request IDs, order numbers, anything that grows with usage.

Building Custom Exporters

The instrumented HTTP server above is one program. The exporter below is a separate binary — it scrapes an external service that doesn't speak Prometheus, and translates its JSON stats into metrics. Most of the systems you operate fall into this category: job queues, legacy services, internal APIs.

A custom exporter implements the prometheus.Collector interface, scrapes the external system at each Prometheus pull, and translates the response into metrics.

// ServiceStats represents the response from our external service
type ServiceStats struct {
	QueueDepth    int     `json:"queue_depth"`
	ProcessedJobs int     `json:"processed_jobs"`
	AvgLatencyMs  float64 `json:"avg_latency_ms"`
	WorkersActive int     `json:"workers_active"`
}

type ServiceCollector struct {
	endpoint     string
	client       *http.Client
	queueDepth   *prometheus.Desc
	jobsTotal    *prometheus.Desc
	latencyMs    *prometheus.Desc
	workersUp    *prometheus.Desc
	scrapeErrors *prometheus.Desc
}

// NewServiceCollector creates a Prometheus Collector that fetches metrics from
// the given endpoint at each scrape. The HTTP client has a 5s timeout.
func NewServiceCollector(endpoint string) *ServiceCollector {
	return &ServiceCollector{
		endpoint: endpoint,
		client:   &http.Client{Timeout: 5 * time.Second},
		queueDepth: prometheus.NewDesc(
			"service_queue_depth",
			"Number of items waiting in the queue.",
			nil, nil,
		),
		jobsTotal: prometheus.NewDesc(
			"service_processed_jobs_total",
			"Total number of processed jobs.",
			nil, nil,
		),
		latencyMs: prometheus.NewDesc(
			"service_avg_latency_milliseconds",
			"Average job processing latency in milliseconds.",
			nil, nil,
		),
		workersUp: prometheus.NewDesc(
			"service_workers_active",
			"Number of active workers.",
			nil, nil,
		),
		scrapeErrors: prometheus.NewDesc(
			"service_scrape_errors_total",
			"Total scrape errors.",
			nil, nil,
		),
	}
}

// Describe sends all metric descriptors to ch. Required by prometheus.Collector.
func (c *ServiceCollector) Describe(ch chan<- *prometheus.Desc) {
	ch <- c.queueDepth
	ch <- c.jobsTotal
	ch <- c.latencyMs
	ch <- c.workersUp
	ch <- c.scrapeErrors
}

// Collect fetches fresh stats from the external service and sends metrics to ch.
// Called by Prometheus on each scrape. Records a scrape error metric on failure.
func (c *ServiceCollector) Collect(ch chan<- prometheus.Metric) {
	resp, err := c.client.Get(c.endpoint + "/stats")
	if err != nil {
		ch <- prometheus.MustNewConstMetric(c.scrapeErrors, prometheus.CounterValue, 1)
		return
	}
	defer resp.Body.Close()

	var stats ServiceStats
	if err := json.NewDecoder(resp.Body).Decode(&stats); err != nil {
		ch <- prometheus.MustNewConstMetric(c.scrapeErrors, prometheus.CounterValue, 1)
		return
	}

	ch <- prometheus.MustNewConstMetric(c.queueDepth, prometheus.GaugeValue, float64(stats.QueueDepth))
	ch <- prometheus.MustNewConstMetric(c.jobsTotal, prometheus.CounterValue, float64(stats.ProcessedJobs))
	ch <- prometheus.MustNewConstMetric(c.latencyMs, prometheus.GaugeValue, stats.AvgLatencyMs)
	ch <- prometheus.MustNewConstMetric(c.workersUp, prometheus.GaugeValue, float64(stats.WorkersActive))
}

func main() {
	collector := NewServiceCollector("http://localhost:9000")
	prometheus.MustRegister(collector)

	http.Handle("/metrics", promhttp.Handler())
	log.Println("Exporter listening on :9101")
	// Output: 2026/07/20 10:00:00 Exporter listening on :9101
	log.Fatal(http.ListenAndServe(":9101", nil))
}

Collect runs every time Prometheus scrapes, usually every 15-30 seconds. Fetch fresh data each time. A stale queue depth is worse than no queue depth because it gives you false confidence.

Gotcha: Always set a Timeout on the HTTP client inside your collector. If the external system hangs and Prometheus scrapes every 15 seconds, a single hanging request cascades into every scrape blocking. Eventually your entire /metrics endpoint stops responding and Prometheus marks your instance as down.

Histograms and Summaries

Counters and gauges answer "how many" and "how much right now." Histograms answer "what does the distribution look like?" You need a histogram to detect that the 99th percentile of your request latency jumped from 200ms to 4 seconds while the average stayed at 50ms.

Histograms count how many observations fall into configurable buckets. Prometheus computes percentiles from those buckets server-side with histogram_quantile():

// Histogram: define buckets, Prometheus calculates percentiles in PromQL
reqDuration := prometheus.NewHistogram(prometheus.HistogramOpts{
	Name:    "request_duration_seconds",
	Buckets: prometheus.DefBuckets, // .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10
})

// Summary: calculates quantiles in your process, higher CPU cost
reqDurationSummary := prometheus.NewSummary(prometheus.SummaryOpts{
	Name:       "request_duration_summary_seconds",
	Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
	MaxAge:     10 * time.Minute,
})

Use histograms by default. If you have 5 replicas, Prometheus can compute the 95th percentile across all of them from histogram buckets. Summaries compute quantiles per instance, and those can't be combined correctly across replicas.

PromQL Basics

Once metrics are in Prometheus, PromQL is how you query them:

# Request rate over 5 minutes
rate(myapp_requests_total[5m])

# Error rate as a percentage
sum(rate(myapp_requests_total{status=~"5.."}[5m]))
/ sum(rate(myapp_requests_total[5m])) * 100

# 95th percentile latency from histogram
histogram_quantile(0.95, rate(myapp_request_duration_seconds_bucket[5m]))

# Queue depth alert threshold
myapp_queue_depth > 500

The rate() function is essential for counters. It computes per-second increase and handles counter resets correctly. When your service restarts and the counter goes back to zero, rate() does not report a negative rate.

Alertmanager Integration

Your Go exporter does not configure alerts. Prometheus evaluates alerting rules and fires to Alertmanager. The separation is intentional: your code just exposes numbers, Prometheus decides when to alert.

groups:
  - name: service_alerts
    rules:
      - alert: QueueBacklog
        expr: myapp_queue_depth > 500
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Queue depth exceeds 500 for 10 minutes"

      - alert: HighErrorRate
        expr: >
          sum(rate(myapp_requests_total{status=~"5.."}[5m]))
          / sum(rate(myapp_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 5 minutes"

The queue alert would have fired at 10 minutes. Not 90.

Running Prometheus Locally

You can run Prometheus locally with Docker to try the queries above against your Go app.

Create prometheus.yml in your working directory:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'myapp'
    static_configs:
      - targets: ['host.docker.internal:8080']

host.docker.internal is how Docker on Mac and Windows reaches your localhost. On Linux, replace it with your machine's LAN IP (e.g. 192.168.1.x:8080).

Start Prometheus:

docker run -d \
  -p 9090:9090 \
  --name prometheus \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Then run the instrumented Go app from the "Instrumenting Your Application" section:

go run main.go
# serving metrics on :8080/metrics

Hit the app a few times to generate traffic:

for i in $(seq 1 20); do curl -s http://localhost:8080/api/data > /dev/null; done

Open http://localhost:9090 and try these queries:

# Total requests
myapp_requests_total

# Request rate per second (last 1 minute)
rate(myapp_requests_total[1m])

# Current queue depth
myapp_queue_depth

# 95th percentile latency
histogram_quantile(0.95, rate(myapp_request_duration_seconds_bucket[1m]))

You'll see your metrics appear in Prometheus's graph UI within 15-30 seconds of the first scrape.

Putting It Together: Instrumentation That Matters

The lesson from the incident at the start: you cannot debug what you cannot see. Adding a queue depth gauge is three lines of Go. The payoff is an alert that fires while you are still asleep, before the queue has grown to 8000.

Wire the exporter into devctl as the metrics subcommand:

var metricsCmd = &cobra.Command{
    Use:   "metrics",
    Short: "Start the Prometheus exporter",
    RunE: func(cmd *cobra.Command, args []string) error {
        collector := NewServiceCollector("http://localhost:9000")
        prometheus.MustRegister(collector)

        http.Handle("/metrics", promhttp.Handler())
        fmt.Println("serving metrics on :9101/metrics")
        // Output: serving metrics on :9101/metrics
        return http.ListenAndServe(":9101", nil)
    },
}

func init() {
    rootCmd.AddCommand(metricsCmd)
}

Run it with:

devctl metrics

The same pattern applies to everything: latency, error rates, cache hit ratios, pending jobs, worker counts. Each metric you skip is a gap where the next incident hides.

Start with the metrics that answer your most likely questions. For most services, you need four things: request rate, error rate, latency as a histogram, and the state of any queues or buffers. Those four tell you whether the service is healthy and where to look if it is not.

Queue Depth Exporter

A custom Prometheus exporter implementing the prometheus.Collector interface. It scrapes a service's JSON stats endpoint at each Prometheus pull and translates queue depth, job counts, latency, and worker status into standard metrics.

Input: An external service endpoint exposing a /stats JSON API. Output: Prometheus metrics on :9101/metrics including queue depth, processed jobs, average latency, and active workers.

Full source: examples/queue-depth-exporter

Key Takeaways

• Prometheus pulls metrics from your /metrics endpoint on a schedule. Your app exposes numbers, Prometheus stores them, Alertmanager notifies.
• Use counters for totals, gauges for current state (queue depth, active connections), histograms for latency distributions.
• Label cardinality kills. User IDs, request IDs, and order numbers as labels create millions of series and will OOM Prometheus. Labels must be bounded sets.
• Custom exporters implement prometheus.Collector and fetch fresh data from external systems at each scrape. Stale values are worse than no values.
• Prefer histograms over summaries. Histograms aggregate across replicas with histogram_quantile(); summaries cannot.
• Always set a Timeout on the HTTP client inside a collector. A slow external system should not block your metrics endpoint.
• Don't use MustRegister in code that runs more than once. Use prometheus.NewRegistry() in tests.

🎁 What if your Go agent could tail log files, parse and enrich every line, and fire a Slack alert the instant error rates spike?