10 - Metrics & Prometheus

🎁 How do you answer "is the service slow?" with a number instead of a guess, and get alerted before users even notice?

Observability starts with numbers. Logs tell you what happened, traces show you the path, but metrics tell you the shape of your system right now: request rates, error counts, latencies, resource usage. Prometheus is the de facto standard for collecting and querying those numbers, and Go has first-class support for it.

Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.

Prometheus Data Model

Every metric in Prometheus is a time series identified by a metric name and a set of key-value labels:

http_requests_total{method="GET", status="200"} 1234

The name describes what's being measured. Labels add dimensions, so you can slice and dice by method, status, endpoint, instance. Each unique combination of name + labels is a separate time series.

Prometheus scrapes targets on a schedule (typically every 15-30 seconds), pulling metrics from an HTTP endpoint. Your application exposes /metrics in a text format that Prometheus understands.

Metric Types

Prometheus defines four core types:

Counter: monotonically increasing value. Use for totals: requests served, errors encountered, bytes transferred. Never decreases (except on restart).
Gauge: value that goes up and down. Use for current state: temperature, active connections, queue depth.
Histogram: samples observations into configurable buckets. Use for latencies and sizes when you need percentiles.
Summary: similar to histogram but calculates quantiles client-side. Use when you need precise quantiles over a single instance.

Instrumenting Go Applications

The prometheus/client_golang library provides everything you need. Install it:

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

Basic instrumentation with all four types:

package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	requestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "myapp_requests_total",
			Help: "Total number of HTTP requests.",
		},
		[]string{"method", "path", "status"},
	)

	activeConnections = prometheus.NewGauge(
		prometheus.GaugeOpts{
			Name: "myapp_active_connections",
			Help: "Number of active connections.",
		},
	)

	requestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "myapp_request_duration_seconds",
			Help:    "Request latency in seconds.",
			Buckets: []float64{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0},
		},
		[]string{"method", "path"},
	)
)

func init() {
	prometheus.MustRegister(requestsTotal, activeConnections, requestDuration)
}

func main() {
	http.Handle("/metrics", promhttp.Handler())
	http.HandleFunc("/api/data", func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		activeConnections.Inc()
		defer activeConnections.Dec()

		// Simulate work
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)

		requestsTotal.WithLabelValues(r.Method, "/api/data", "200").Inc()
		requestDuration.WithLabelValues(r.Method, "/api/data").Observe(time.Since(start).Seconds())

		w.Write([]byte(`{"status":"ok"}`))
	})

	fmt.Println("serving metrics on :8080/metrics")
	// Output: serving metrics on :8080/metrics
	http.ListenAndServe(":8080", nil)
}

Hit /metrics and you'll see output like:

# HELP myapp_requests_total Total number of HTTP requests.
# TYPE myapp_requests_total counter
myapp_requests_total{method="GET",path="/api/data",status="200"} 42

Building Custom Exporters

An exporter collects metrics from an external system and translates them into Prometheus format. This is how you monitor things that don't natively speak Prometheus: databases, message queues, legacy services.

The pattern: implement the prometheus.Collector interface, scrape the external system in the Collect method, and register your collector.

Here's a complete exporter that collects metrics from a hypothetical service API:

package main

import (
	"encoding/json"
	"log"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// ServiceStats represents the response from our external service
type ServiceStats struct {
	QueueDepth    int     `json:"queue_depth"`
	ProcessedJobs int     `json:"processed_jobs"`
	AvgLatencyMs  float64 `json:"avg_latency_ms"`
	WorkersActive int     `json:"workers_active"`
}

type ServiceCollector struct {
	endpoint     string
	client       *http.Client
	queueDepth   *prometheus.Desc
	jobsTotal    *prometheus.Desc
	latency      *prometheus.Desc
	workersUp    *prometheus.Desc
	scrapeErrors *prometheus.Desc
}

func NewServiceCollector(endpoint string) *ServiceCollector {
	return &ServiceCollector{
		endpoint: endpoint,
		client:   &http.Client{Timeout: 5 * time.Second},
		queueDepth: prometheus.NewDesc(
			"service_queue_depth",
			"Number of items waiting in the queue.",
			nil, nil,
		),
		jobsTotal: prometheus.NewDesc(
			"service_processed_jobs_total",
			"Total number of processed jobs.",
			nil, nil,
		),
		latency: prometheus.NewDesc(
			"service_avg_latency_seconds",
			"Average processing latency in seconds.",
			nil, nil,
		),
		workersUp: prometheus.NewDesc(
			"service_workers_active",
			"Number of active workers.",
			nil, nil,
		),
		scrapeErrors: prometheus.NewDesc(
			"service_scrape_errors_total",
			"Total scrape errors.",
			nil, nil,
		),
	}
}

func (c *ServiceCollector) Describe(ch chan<- *prometheus.Desc) {
	ch <- c.queueDepth
	ch <- c.jobsTotal
	ch <- c.latency
	ch <- c.workersUp
	ch <- c.scrapeErrors
}

func (c *ServiceCollector) Collect(ch chan<- prometheus.Metric) {
	resp, err := c.client.Get(c.endpoint + "/stats")
	if err != nil {
		ch <- prometheus.MustNewConstMetric(c.scrapeErrors, prometheus.CounterValue, 1)
		return
	}
	defer resp.Body.Close()

	var stats ServiceStats
	if err := json.NewDecoder(resp.Body).Decode(&stats); err != nil {
		ch <- prometheus.MustNewConstMetric(c.scrapeErrors, prometheus.CounterValue, 1)
		return
	}

	ch <- prometheus.MustNewConstMetric(c.queueDepth, prometheus.GaugeValue, float64(stats.QueueDepth))
	ch <- prometheus.MustNewConstMetric(c.jobsTotal, prometheus.CounterValue, float64(stats.ProcessedJobs))
	ch <- prometheus.MustNewConstMetric(c.latency, prometheus.GaugeValue, stats.AvgLatencyMs/1000.0)
	ch <- prometheus.MustNewConstMetric(c.workersUp, prometheus.GaugeValue, float64(stats.WorkersActive))
}

func main() {
	collector := NewServiceCollector("http://localhost:9000")
	prometheus.MustRegister(collector)

	http.Handle("/metrics", promhttp.Handler())
	log.Println("Exporter listening on :9101")
	// Output: 2026/07/20 10:00:00 Exporter listening on :9101
	log.Fatal(http.ListenAndServe(":9101", nil))
}

Note that Collect is called every time Prometheus scrapes. You fetch fresh data from the external system at scrape time. No caching stale values.

Histograms & Summaries

Both track distributions, but they differ in where computation happens:

	Histogram	Summary
Quantile calculation	Server-side (PromQL)	Client-side
Aggregation across instances	Yes	No
Bucket configuration	Required	Optional (quantiles)
Cost	Low per observation	Higher (maintains sliding window)

Use histograms when you need to aggregate across instances or want flexible queries. This is the default choice for most DevOps use cases.

Use summaries only when you need precise quantiles from a single instance and can't tolerate the approximation of histogram bucket boundaries.

// Histogram: define buckets, Prometheus calculates percentiles
reqDuration := prometheus.NewHistogram(prometheus.HistogramOpts{
	Name:    "request_duration_seconds",
	Buckets: prometheus.DefBuckets, // .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10
})

// Summary: calculate quantiles client-side
reqDurationSummary := prometheus.NewSummary(prometheus.SummaryOpts{
	Name:       "request_duration_summary_seconds",
	Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
	MaxAge:     10 * time.Minute,
})

PromQL Basics

Once metrics are in Prometheus, you query them with PromQL:

# Request rate over 5 minutes
rate(myapp_requests_total[5m])

# Error rate as a percentage
sum(rate(myapp_requests_total{status=~"5.."}[5m]))
/ sum(rate(myapp_requests_total[5m])) * 100

# 95th percentile latency from histogram
histogram_quantile(0.95, rate(myapp_request_duration_seconds_bucket[5m]))

# Active connections per instance
myapp_active_connections

# Queue depth alert threshold
service_queue_depth > 100

The rate() function is essential for counters. It calculates per-second increase, handling counter resets correctly.

Alertmanager Integration

Prometheus evaluates alerting rules and fires alerts to Alertmanager. Define rules in Prometheus config:

groups:
  - name: service_alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(myapp_requests_total{status=~"5.."}[5m])) / sum(rate(myapp_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 5 minutes"

      - alert: QueueBacklog
        expr: service_queue_depth > 500
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Queue depth exceeds 500 for 10 minutes"

Your Go exporter doesn't need to know about alerts. It just exposes metrics. Prometheus and Alertmanager handle the rest, keeping your instrumentation code simple.

Queue Depth Exporter

A custom Prometheus exporter implementing the prometheus.Collector interface. It scrapes a service's JSON stats endpoint at each Prometheus pull and translates queue depth, job counts, latency, and worker status into standard metrics.

Input: An external service endpoint exposing a /stats JSON API. Output: Prometheus metrics on :9101/metrics including queue depth, processed jobs, average latency, and active workers.

Full source: examples/queue-depth-exporter

Key Takeaways

Prometheus pulls metrics from your application's /metrics endpoint on a schedule. Your app is a passive target.
Use counters for totals, gauges for current state, histograms for latency distributions.
Custom exporters implement prometheus.Collector and fetch data from external systems at scrape time.
Prefer histograms over summaries. Histograms aggregate across instances and give you flexible PromQL queries.
Keep label cardinality low. High-cardinality labels (user IDs, request IDs) will blow up your storage.
The exporter pattern separates concerns: your exporter translates, Prometheus stores, Alertmanager notifies.

🎁 What if your Go agent could tail log files, parse and enrich every line, and fire a Slack alert the instant error rates spike?