04 - Pipes, Redirection & IPC

🎁 Two programs on the same machine need to exchange a stream of bytes. They don't share memory and they can't touch each other's variables. So what's the plumbing that lets one hand data to another, the same plumbing the shell uses for cat file | grep error?

Inter-process communication (IPC) is how separate processes talk. There are several mechanisms: shared files, signals, shared memory, pipes, and sockets. This lesson covers the two you'll reach for most in Go: pipes and Unix domain sockets.

Kernel Pipes with os.Pipe

os.Pipe creates a one-directional channel in the kernel and hands you both ends as *os.File values. Whatever you write to the write end becomes readable from the read end:

r, w, err := os.Pipe()
if err != nil {
    log.Fatal(err)
}

go func() {
    fmt.Fprint(w, "hello through the pipe")
    w.Close()   // reader sees EOF once the write end closes
}()

buf, _ := io.ReadAll(r)
fmt.Println(string(buf))   // hello through the pipe

Both ends are real file descriptors (recall from The OS Interface that everything is an fd). Closing the write end is what lets the reader see io.EOF. If you forget to close it, the reader blocks forever waiting for more.

Standard Streams as File Descriptors

os.Stdin, os.Stdout, and os.Stderr are just *os.File values wrapping descriptors 0, 1, and 2. Because a pipe end is also an *os.File, you can splice a pipe into a program's standard streams. That's the whole trick behind redirection.

r, w, _ := os.Pipe()
os.Stdout = w              // redirect this program's stdout into the pipe

fmt.Println("captured!")   // goes into the pipe, not the terminal
w.Close()

line, _ := bufio.NewReader(r).ReadString('\n')
fmt.Fprint(os.Stderr, "got: "+line)   // got: captured!

Reassigning the global os.Stdout like this is handy for a demonstration, but in real code you'd pass a pipe end to whatever needs it rather than mutating a global. The same idea lets you feed one end of a pipe to a child process as its stdout, which is how you capture a command's output. We wire that up properly with os/exec in Running Processes.

In-Memory Pipes with io.Pipe

Sometimes you need to connect an API that writes to one that reads, without any kernel involvement or file descriptor. io.Pipe gives you an in-memory, synchronous pipe: an io.PipeReader and io.PipeWriter where each Write blocks until a Read consumes it.

pr, pw := io.Pipe()

go func() {
    enc := json.NewEncoder(pw)     // an API that needs an io.Writer
    enc.Encode(map[string]int{"count": 3})
    pw.Close()
}()

var out map[string]int
json.NewDecoder(pr).Decode(&out)   // an API that needs an io.Reader
fmt.Println(out["count"])          // 3

The distinction: reach for io.Pipe to glue Writer-shaped and Reader-shaped code together inside one process. Reach for os.Pipe when a real descriptor has to cross a process boundary.

Unix Domain Sockets

A Unix domain socket is a bidirectional channel between processes on the same host, addressed by a path on the filesystem instead of an IP and port. You create one exactly like a TCP server, but with the network "unix":

os.Remove("/tmp/app.sock")   // clear any stale socket file first
ln, err := net.Listen("unix", "/tmp/app.sock")
if err != nil {
    log.Fatal(err)
}
defer ln.Close()

conn, _ := ln.Accept()
io.Copy(conn, conn)   // echo everything back

A client connects with net.Dial and the same address:

conn, err := net.Dial("unix", "/tmp/app.sock")
if err != nil {
    log.Fatal(err)
}
fmt.Fprintln(conn, "ping")

reply, _ := bufio.NewReader(conn).ReadString('\n')
fmt.Print(reply)   // ping

Two things to remember. The socket is a file on disk, so remove a stale one before Listen and set its permissions to control who may connect. And because there's no IP stack involved, a Unix socket is faster than talking to localhost over TCP, which makes it the natural choice for a local control channel (the capstone uses one for exactly this).

Channels Versus Pipes and Sockets

It's easy to confuse Go channels with pipes, because both move data between concurrent things. The difference is the boundary. A channel moves Go values between goroutines inside one process; it vanishes when the process exits and can't reach anything outside it.

ch := make(chan string)         // in-process only
go func() { ch <- "in-memory" }()
fmt.Println(<-ch)               // in-memory

A pipe or socket moves raw bytes across a process boundary, through the kernel, so a completely separate program can be on the other end. When you need two goroutines to cooperate, use a channel (see Channel Patterns). When you need two programs to cooperate, use a pipe or a socket.

Key Takeaways

• os.Pipe() returns (r, w *os.File, err), a real kernel pipe; write to w, read from r, and close w so the reader sees io.EOF.
• os.Stdin/Stdout/Stderr are *os.File on descriptors 0/1/2, so you can redirect them by assigning a pipe end, and hand a pipe end to a child as its stdout.
• io.Pipe() is an in-memory, synchronous pipe (io.PipeReader/io.PipeWriter) with no descriptor; use it to connect a Writer-based API to a Reader-based one in-process.
• Unix domain sockets (net.Listen("unix", path) / net.Dial("unix", path)) are bidirectional, addressed by a filesystem path, faster than TCP loopback; remove stale socket files and set their permissions.
• Channels move Go values within one process; pipes and sockets move bytes across process boundaries. Pick by whether the other side is a goroutine or a separate program.

🎁 You can talk to another process now, but what if you need to start that process yourself, feed it input, capture its output, and know whether it succeeded? How does Go run git status or ffmpeg and hand you the result?