14 - Capstone: Monitoring Daemon

🎁 You've learned files, processes, signals, sockets, privileges, and daemons as separate pieces. What happens when you wire them into one real program: a daemon that watches a directory, guards its own state, reloads on demand, and answers questions over a socket, all at once?

In this capstone you'll build watchd, a small monitoring daemon. It watches a directory for changes, prevents a second copy of itself from running, reloads its config on SIGHUP, shuts down gracefully, and exposes live stats over a Unix domain socket. Everything comes from the standard library, and every section maps to a lesson you've already done.

Project Layout

watchd is small enough to live in a single main.go. It has a config, a watcher that holds the running state, a stats socket, and a main that wires them together under one shutdown context.

package main

import (
    "context"
    "flag"
    "fmt"
    "log/slog"
    "net"
    "os"
    "os/signal"
    "sync"
    "sync/atomic"
    "syscall"
    "time"
)

Those imports are the whole toolbox from this course: os for files, net for the socket, os/signal for lifecycle, sync/atomic for safe shared state. We'll fill in each piece, then assemble main at the end.

Config and Flags

Configuration comes from flags, following the CLI pattern from Configuration. Keep it in a struct so a reload can swap the whole thing atomically:

type Config struct {
    Dir      string
    Sock     string
    Interval time.Duration
}

func loadConfig() Config {
    dir := flag.String("dir", ".", "directory to watch")
    sock := flag.String("sock", "/tmp/watchd.sock", "stats socket path")
    every := flag.Duration("interval", 2*time.Second, "poll interval")
    flag.Parse()
    return Config{Dir: *dir, Sock: *sock, Interval: *every}
}

A real service would re-read a file on reload; here we keep it simple and re-parse the same source. What matters is the shape: config is a value you can replace wholesale.

The PID Lock File

Before doing anything, watchd makes sure it's the only instance running. It creates a PID file with O_EXCL, the exclusive-create trick from File I/O Deep and Writing a Daemon:

func lockPID(path string) error {
    f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0644)
    if err != nil {
        return fmt.Errorf("already running? %w", err)
    }
    fmt.Fprintln(f, os.Getpid())
    return f.Close()
}

If the file exists, another watchd holds it and this one refuses to start. We remove the file on shutdown so the next run starts clean.

The Directory Watcher

The heart of watchd is a poller. On a time.Ticker, it lists the directory with os.ReadDir and compares each file's size and modification time against a snapshot, logging anything added, changed, or removed. This is deliberately stdlib-only; the production upgrade is the third-party fsnotify, which uses kernel inotify events instead of polling. inotify is a Linux feature that pushes a notification the instant a file changes, so you skip the repeated directory scans polling does.

type watcher struct {
    logger  *slog.Logger
    changes atomic.Int64             // live stat, read by the socket
    started time.Time
    snap    map[string]int64         // name -> modtime unix
}

func (w *watcher) scan(dir string) {
    entries, err := os.ReadDir(dir)
    if err != nil {
        w.logger.Error("readdir failed", "err", err)
        return
    }
    seen := make(map[string]int64, len(entries))
    for _, e := range entries {
        info, err := e.Info()
        if err != nil {
            continue
        }
        mod := info.ModTime().Unix()
        seen[e.Name()] = mod
        if old, ok := w.snap[e.Name()]; !ok || old != mod {
            w.logger.Info("changed", "file", e.Name())
            w.changes.Add(1)
        }
    }
    w.snap = seen
}

changes is an atomic.Int64 because the stats socket reads it from another goroutine while the watcher writes it. That's the sync & atomic discipline: shared counters need atomic access. The watcher loop itself just ticks until the shutdown context is done:

func (w *watcher) run(ctx context.Context, cfg Config) {
    tick := time.NewTicker(cfg.Interval)
    defer tick.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-tick.C:
            w.scan(cfg.Dir)
        }
    }
}

Stats over a Unix Socket

A local operator should be able to ask watchd how it's doing without stopping it. We expose a read-only stats endpoint over a Unix domain socket, the IPC channel from Pipes & IPC. Each connection gets a one-line status report:

func (w *watcher) serveStats(ctx context.Context, sockPath string) {
    os.Remove(sockPath)                        // clear any stale socket
    ln, err := net.Listen("unix", sockPath)
    if err != nil {
        w.logger.Error("stats listen failed", "err", err)
        return
    }
    go func() { <-ctx.Done(); ln.Close() }()   // unblock Accept on shutdown

    for {
        conn, err := ln.Accept()
        if err != nil {
            return                             // listener closed: we're done
        }
        uptime := time.Since(w.started).Round(time.Second)
        fmt.Fprintf(conn, "uptime=%s changes=%d files=%d\n",
            uptime, w.changes.Load(), len(w.snap))
        conn.Close()
    }
}

Now nc -U /tmp/watchd.sock (netcat in Unix-socket mode, a quick way to connect to the socket and read a line) prints something like uptime=1m30s changes=4 files=12. Closing the listener when the context is done makes the blocked Accept return, so this goroutine exits cleanly with the rest.

Signals: Reload and Shutdown

Two signal concerns, straight from Signals & Lifecycle. SIGINT/SIGTERM trigger graceful shutdown through a context; SIGHUP triggers a config reload in its own goroutine so a reload never stops the service:

func watchReload(logger *slog.Logger, cur *atomic.Pointer[Config]) {
    hup := make(chan os.Signal, 1)
    signal.Notify(hup, syscall.SIGHUP)
    go func() {
        for range hup {
            cfg := loadConfig()
            cur.Store(&cfg)                    // swap config atomically
            logger.Info("config reloaded", "dir", cfg.Dir)
        }
    }()
}

The shutdown side uses signal.NotifyContext, so one cancelled context stops the watcher loop and the stats server together.

Wiring main

Here's the whole program assembled. main locks the PID file, builds the watcher, starts the stats server and reload handler, and blocks until a stop signal drains everything:

func main() {
    cfg := loadConfig()
    logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))

    if err := lockPID("/tmp/watchd.pid"); err != nil {
        logger.Error("cannot start", "err", err)
        os.Exit(1)
    }
    defer os.Remove("/tmp/watchd.pid")

    ctx, stop := signal.NotifyContext(context.Background(),
        syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    w := &watcher{logger: logger, started: time.Now(), snap: map[string]int64{}}

    var wg sync.WaitGroup
    wg.Add(2)
    go func() { defer wg.Done(); w.run(ctx, cfg) }()
    go func() { defer wg.Done(); w.serveStats(ctx, cfg.Sock) }()

    logger.Info("watchd started", "pid", os.Getpid(), "dir", cfg.Dir)
    <-ctx.Done()
    logger.Info("shutting down")
    wg.Wait()                                  // let both goroutines finish
    os.Remove(cfg.Sock)
}

The sync.WaitGroup ensures shutdown waits for both goroutines to return before main exits, so the socket file and PID file are removed only after everything has stopped.

Running It

Build it and run it against any directory, unprivileged (in production you'd set User= in a systemd unit, from Writing a Daemon):

go build -o watchd .
./watchd -dir /tmp/inbox -interval 1s
# {"time":"...","level":"INFO","msg":"watchd started","pid":50122,"dir":"/tmp/inbox"}

Touch a file in the watched directory and a changed line appears in the log. Ask for stats over the socket while it runs, and send SIGHUP to reload or Ctrl+C to stop:

nc -U /tmp/watchd.sock        # uptime=12s changes=1 files=3
kill -HUP  $(cat /tmp/watchd.pid)   # reload config
kill -TERM $(cat /tmp/watchd.pid)   # graceful shutdown

That's a complete systems program: it opens files, guards a resource, speaks a protocol over a socket, handles signals, and cleans up after itself.

Key Takeaways

• Config as a value: parse flags into a Config struct you can swap wholesale on reload.
• Single instance: a PID file created with O_CREATE|O_EXCL refuses a second start; remove it on shutdown.
• Directory watching: a time.Ticker plus os.ReadDir and a modtime snapshot detects changes with only the standard library (fsnotify is the production upgrade).
• Safe shared state: the change counter is an atomic.Int64 because the watcher and the stats server touch it concurrently.
• Local control channel: a Unix domain socket serves live stats; closing the listener on ctx.Done() unblocks Accept for clean shutdown.
• Lifecycle: SIGHUP reloads config in its own goroutine; signal.NotifyContext cancels one context that stops every goroutine, and a sync.WaitGroup drains them before exit.

You just built a real daemon from the ground up, using nothing but Go's standard library and the operating system underneath it. Every piece, the file descriptors, the socket, the signals, the privilege model, came from a specific lesson in this course, and now they run together in one program you can actually deploy. That's systems programming: not magic, just the OS interface used well. Go build something with it.