13 - Writing a Daemon

🎁 A daemon runs for months without a terminal, restarts itself when it crashes, and reloads its config without dropping a single connection. The old way to build one involved a scary fork-twice ritual. On a modern Linux box, almost all of that is somebody else's job now. Whose?

A daemon is a long-running background service: a web server, a job runner, the monitor you'll build next lesson. This lesson is about making a Go program behave like one, the current way. That means writing a plain foreground program and letting the init system handle backgrounding, restarts, and logging, while your code handles reloads, shutdown, and its own identity.

Foreground, Not Double-Fork

Classic Unix daemons detached from the terminal by forking twice, calling setsid, and redirecting their own streams. On a systemd-managed system you should not do any of that. The init system expects your program to run in the foreground and it handles the backgrounding for you.

// The wrong instinct on modern Linux: manual daemonization.
// You do NOT need setsid, double-fork, or redirecting your own stdio.
func main() {
    run() // just run in the foreground; let systemd background you
}

This is simpler and more reliable. systemd tracks your process, restarts it on failure, captures its output, and manages its lifecycle. Your job is to be a well-behaved foreground program.

A systemd Unit

You describe the service to systemd with a unit file, typically at /etc/systemd/system/monitor.service. This is where backgrounding, restart policy, and identity live, outside your Go code:

[Unit]
Description=Monitor daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/monitor -config /etc/monitor.yaml
Restart=on-failure
User=monitor
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Restart=on-failure gives you crash recovery for free. User=monitor drops privileges without a line of Go, which is cleaner than the Setuid dance from the permissions lesson. ExecReload wires systemctl reload monitor to a SIGHUP, which your program will handle.

Logging to Standard Out

A foreground daemon doesn't manage log files. It writes to stdout and stderr, and systemd's journal captures every line with timestamps and metadata. Use structured logging so those lines are machine-readable:

import "log/slog"

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Info("service started", "pid", os.Getpid())
// {"time":"...","level":"INFO","msg":"service started","pid":48213}

This is the same slog from Go in Practice. You get log rotation, retention, and journalctl -u monitor querying without writing any of it, because logging to stdout hands the whole problem to the platform. Log rotation is automatically archiving and deleting old log files so they don't grow forever and fill the disk.

PID Files and Single-Instance

systemd tracks your PID for you, so you rarely need a PID file. But when you run outside systemd, or want to guarantee only one instance starts, a PID file created with O_EXCL is the tool. It reuses the exclusive-create trick from the file I/O lesson:

f, err := os.OpenFile("/run/monitor.pid",
    os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0644)
if errors.Is(err, os.ErrExist) {
    log.Fatal("already running")
}
fmt.Fprintln(f, os.Getpid())
f.Close()
defer os.Remove("/run/monitor.pid")

If the file already exists, another instance holds it, so you refuse to start. This simple version has one weakness: a crash leaves a stale PID file behind. A file lock (flock) is the sturdier alternative, since the kernel releases it automatically when the process dies, but the O_EXCL file is enough for most services.

Reloading Config on SIGHUP

The convention is that SIGHUP means "reload your configuration without restarting." Handle it separately from the shutdown signals, using the signal handling from Lesson 6:

hup := make(chan os.Signal, 1)
signal.Notify(hup, syscall.SIGHUP)
go func() {
    for range hup {
        cfg, err := loadConfig(*configPath)
        if err != nil {
            logger.Error("reload failed, keeping old config", "err", err)
            continue
        }
        current.Store(&cfg) // swap in the new config atomically
        logger.Info("config reloaded")
    }
}()

The key detail: if the reload fails, keep running with the old config. A daemon should never fall over because someone saved a typo in a config file.

Graceful Shutdown

Finally, a daemon must stop cleanly. When systemd sends SIGTERM, you want to stop accepting new work, finish what's in flight, and exit. signal.NotifyContext turns those signals into a cancellable context:

ctx, stop := signal.NotifyContext(context.Background(),
    syscall.SIGINT, syscall.SIGTERM)
defer stop()

srv := startServer(ctx)
<-ctx.Done()                 // blocks until a stop signal arrives
logger.Info("shutting down")

shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx)    // drain in-flight work, up to 10s

This is the graceful shutdown pattern, now the backbone of a real service. Put the pieces together (foreground process, structured logs, SIGHUP reload, SIGTERM drain) and you have a daemon that a platform can run and operators can trust.

Key Takeaways

• On modern Linux, don't manually daemonize (no double-fork or setsid). Run in the foreground and let systemd handle backgrounding, restarts, and capture.
• A systemd unit file owns backgrounding, Restart=on-failure, User= for privilege drop, and ExecReload to map systemctl reload onto SIGHUP.
• Log structured JSON to stdout with slog; the journal captures it. Query with journalctl, and let the platform handle rotation and retention.
• systemd tracks your PID, so a PID file is optional; when needed, create it with O_EXCL to enforce a single instance (a flock-based lock is sturdier, since it releases automatically on a crash).
• Handle SIGHUP to reload config without restarting, and keep the old config if the reload fails.
• Use signal.NotifyContext(ctx, SIGINT, SIGTERM) for graceful shutdown: stop new work, drain in-flight work with a timeout, then exit.

🎁 You've learned files, processes, signals, sockets, privileges, and daemons as separate pieces. What happens when you wire them all into one real program: a daemon that watches a directory, locks its state, reloads on demand, and answers questions over a socket, all at once?