02 - SSH & Remote Execution
📋 Jump to Takeaways🎁 What if you could run commands on remote machines, transfer files, and tunnel through bastion hosts, all from native Go code without ever shelling out to the ssh binary?
Last lesson you wired devctl deploy and devctl status into the Cobra tree. Now the team needs two more subcommands: devctl ssh run <host> <command> to execute commands on the fleet, and devctl ssh upload to push config files before switching traffic.
Your team needs to roll out a config change to 50 servers. You write the obvious bash loop:
for host in $(cat hosts.txt); do
ssh -o ConnectTimeout=5 deployer@$host "sudo systemctl restart app && systemctl is-active app"
doneIt works on the first run. Then one host has a stale SSH key in known_hosts and the whole loop stops because ssh prompts for confirmation. Another host is unreachable and ssh hangs for 30 seconds before timing out, holding up every host after it. Two hosts restart successfully, three don't, and the only way to know which is to scroll through the mixed stdout from all 50 connections.
You add -o BatchMode=yes to skip prompts. You add -o ConnectTimeout=10. You redirect per-host output to files. The script is now 40 lines of bash plumbing and it's still sequential. Running it against 50 hosts takes 10 minutes, most of which is waiting.
The SSH library solves this. It gives you a real Go client: typed errors you can inspect, parallel execution with goroutines, per-host timeouts via context.Context, and structured results you can aggregate, filter, and report on.
Imports in code snippets are trimmed for brevity. See the full example linked at the end for complete, compilable source.
Why Not Just Shell Out to ssh?
exec.Command("ssh", host, command) gets you a subprocess. Combined stdout/stderr, a raw exit code, and nothing else.
The SSH library gives you:
- Typed connection errors. A connection refused, a timeout, and a bad host key are three different error types. With a subprocess you get exit code 255 and a string in stderr.
- Parallel execution. Sessions are goroutine-safe. Open 50 sessions concurrently over independent connections, collect results into a channel, and you're done in seconds instead of minutes.
- Cancellation. Every dial and session accepts a
context.Context. Cancel the context and all in-flight operations stop cleanly. - Separate stdout and stderr. Wire each stream to its own buffer. You know exactly what the command wrote where.
- SFTP on the same connection. File transfers over the same authenticated session, no second tool needed.
go get golang.org/x/crypto/ssh@latest
go get github.com/pkg/sftp@latestConnecting to a Host
The core types are ssh.ClientConfig (auth and options) and ssh.Client (a live connection you reuse). Building the config right matters because getting it wrong produces errors that are easy to misread:
package main
import (
"os"
"time"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
)
// newSSHConfig builds an ssh.ClientConfig using key-based auth and host
// verification from ~/.ssh/known_hosts. Timeout covers the TCP dial only.
func newSSHConfig(user, keyPath string) (*ssh.ClientConfig, error) {
key, err := os.ReadFile(keyPath)
if err != nil {
return nil, fmt.Errorf("read key %s: %w", keyPath, err)
}
signer, err := ssh.ParsePrivateKey(key)
if err != nil {
return nil, fmt.Errorf("parse key: %w", err)
}
hostKeyCallback, err := knownhosts.New(os.ExpandEnv("$HOME/.ssh/known_hosts"))
if err != nil {
return nil, fmt.Errorf("load known_hosts: %w", err)
}
return &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{ssh.PublicKeys(signer)},
HostKeyCallback: hostKeyCallback,
Timeout: 10 * time.Second,
}, nil
}The Timeout on ClientConfig is only the TCP dial timeout. If the host accepts the connection but the SSH handshake hangs (common with misconfigured servers), Timeout won't help you. For full handshake protection, wrap the call in a context with a deadline.
For development against hosts not in known_hosts, you can use ssh.InsecureIgnoreHostKey(). Never use it in production automation. Man-in-the-middle attacks against deployment pipelines are a real threat, and InsecureIgnoreHostKey gives you no protection at all.
Once you have a config, dial the host to get an ssh.Client:
// connect dials host:22 and returns a reusable ssh.Client.
// The caller must call client.Close() when done.
func connect(host, user, keyPath string) (*ssh.Client, error) {
config, err := newSSHConfig(user, keyPath)
if err != nil {
return nil, err
}
client, err := ssh.Dial("tcp", host+":22", config)
if err != nil {
return nil, fmt.Errorf("dial %s: %w", host, err)
}
return client, nil
}This client is what you pass to every subsequent operation — running commands, transferring files, opening sessions. One client per host, reused for everything.
Running a Command
Each command runs in a Session. One ssh.Client can open many sessions over the same TCP connection, which is how you avoid the overhead of a full handshake per command:
// runCommand runs a single command on the remote host and returns the combined
// stdout+stderr output. Each call opens a new session; sessions are single-use.
func runCommand(client *ssh.Client, command string) (string, error) {
session, err := client.NewSession()
if err != nil {
return "", fmt.Errorf("new session: %w", err)
}
defer session.Close()
output, err := session.CombinedOutput(command)
if err != nil {
return string(output), fmt.Errorf("command failed: %w\noutput: %s", err, output)
}
return string(output), nil
}CombinedOutput captures stdout and stderr merged together, like 2>&1 in bash. For most diagnostic commands that's fine. When you need to know whether a line came from stdout or stderr, use separate buffers:
// runSeparateStreams runs a command and returns stdout and stderr as distinct
// strings, so callers can tell which stream each line came from.
func runSeparateStreams(client *ssh.Client, command string) (stdout, stderr string, err error) {
session, err := client.NewSession()
if err != nil {
return "", "", err
}
defer session.Close()
var outBuf, errBuf bytes.Buffer
session.Stdout = &outBuf
session.Stderr = &errBuf
err = session.Run(command)
return outBuf.String(), errBuf.String(), err
}Each Session runs exactly one command. You cannot reuse a session once Run, CombinedOutput, or Start has been called. Open a new session for each command. The TCP connection stays open; only the session is single-use.
Command Timeouts
A remote host that becomes unresponsive after accepting the connection will hang your session indefinitely. The dial timeout won't help because the connection is already established. Protect against this with a context:
// runWithTimeout runs a command and kills it with SIGKILL if it exceeds timeout.
// Use this when a remote host may become unresponsive after the connection is established.
func runWithTimeout(client *ssh.Client, command string, timeout time.Duration) (string, error) {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
done := make(chan struct{})
var output []byte
var runErr error
session, err := client.NewSession()
if err != nil {
return "", err
}
defer session.Close()
go func() {
output, runErr = session.CombinedOutput(command)
close(done)
}()
select {
case <-done:
return string(output), runErr
case <-ctx.Done():
session.Signal(ssh.SIGKILL)
return "", fmt.Errorf("command timed out after %v", timeout)
}
}For long-running automation, add keepalives to detect dead connections before they block:
// keepAlive sends periodic keepalive requests to detect dead connections early.
// It returns when the connection fails or done is closed.
func keepAlive(client *ssh.Client, interval time.Duration, done <-chan struct{}) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
_, _, err := client.SendRequest("[email protected]", true, nil)
if err != nil {
return // connection is gone
}
case <-done:
return
}
}
}Running on Many Hosts in Parallel
Sequential execution is the main reason the bash loop was slow. With goroutines, you run all 50 hosts concurrently:
type HostResult struct {
Host string
Output string
Err error
}
// runOnFleet dials all hosts concurrently, runs command on each, and returns
// one HostResult per host. Failed hosts are included with Err set, not skipped.
func runOnFleet(hosts []string, user, keyPath, command string) []HostResult {
results := make(chan HostResult, len(hosts))
for _, host := range hosts {
host := host // capture for goroutine
go func() {
config, err := newSSHConfig(user, keyPath)
if err != nil {
results <- HostResult{Host: host, Err: err}
return
}
client, err := ssh.Dial("tcp", host+":22", config)
if err != nil {
results <- HostResult{Host: host, Err: fmt.Errorf("dial: %w", err)}
return
}
defer client.Close()
out, err := runCommand(client, command)
results <- HostResult{Host: host, Output: out, Err: err}
}()
}
collected := make([]HostResult, 0, len(hosts))
for range hosts {
collected = append(collected, <-results)
}
return collected
}The gotcha here is bounding concurrency. Running 50 goroutines is fine. Running 5,000 isn't, because each goroutine tries to open a TCP connection, and your OS has a limit on open file descriptors. For large fleets, use a semaphore to cap concurrent connections:
sem := make(chan struct{}, 20) // max 20 concurrent SSH connections
go func() {
sem <- struct{}{} // acquire
defer func() { <-sem }() // release
// ... dial and run
}()Without the cap, automation against a large fleet will start failing with "too many open files" and you'll spend an hour wondering why it works for 100 hosts but not 500.
Putting It Together: devctl ssh
Here's what the bash loop from the intro looks like when replaced with Go, wired into devctl as an ssh subcommand. It dials all hosts in parallel, runs a command on each, collects structured results, and reports which hosts failed and why:
// runFleetCommand runs command on all hosts concurrently and prints a per-host
// OK/FAIL summary. Returns an error listing every host that failed.
func runFleetCommand(hosts []string, user, keyPath, command string) error {
results := runOnFleet(hosts, user, keyPath, command)
var failed []string
for _, r := range results {
if r.Err != nil {
fmt.Fprintf(os.Stderr, "[FAIL] %s: %v\n", r.Host, r.Err)
failed = append(failed, r.Host)
} else {
fmt.Printf("[OK] %s: %s\n", r.Host, strings.TrimSpace(r.Output))
}
}
if len(failed) > 0 {
return fmt.Errorf("%d/%d hosts failed: %v", len(failed), len(hosts), failed)
}
return nil
}The bash loop ran sequentially and stopped on the first error. This runs all hosts concurrently, collects every result, and returns a single structured error with the full failure list. The caller knows exactly which hosts succeeded and which didn't.
Wire runFleetCommand into devctl the same way deploy was registered in lesson 01:
var sshCmd = &cobra.Command{
Use: "ssh",
Short: "Run commands on remote hosts",
}
var sshRunCmd = &cobra.Command{
Use: "run <host> <command>",
Short: "Run a command on a single remote host",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
user, _ := cmd.Flags().GetString("user")
key, _ := cmd.Flags().GetString("key")
return runFleetCommand([]string{args[0]}, user, key, args[1])
},
}
func init() {
sshCmd.PersistentFlags().StringP("user", "u", "deployer", "SSH user")
sshCmd.PersistentFlags().StringP("key", "k", "~/.ssh/id_ed25519", "Path to private key")
sshCmd.AddCommand(sshRunCmd)
rootCmd.AddCommand(sshCmd)
}Running devctl ssh run web-01 "systemctl is-active app" now uses the same Cobra flag inheritance and error-exit contract as every other subcommand.
File Transfer with SFTP
Once you have an ssh.Client from connect(), you pass it directly to sftp.NewClient. No second dial, no new authentication — SFTP runs as a subsystem over the same connection:
client, _ := connect(host, user, keyPath)
sc, err := sftp.NewClient(client)
// sc is now an SFTP client on the same SSH connectionWith sc in hand, you can upload, download, and manage files on the remote host:
import "github.com/pkg/sftp"
// uploadFile copies a local file to remotePath over SFTP. The remote directory
// must already exist; call sc.MkdirAll first if it may not.
func uploadFile(client *ssh.Client, localPath, remotePath string) error {
sc, err := sftp.NewClient(client)
if err != nil {
return fmt.Errorf("sftp client: %w", err)
}
defer sc.Close()
src, err := os.Open(localPath)
if err != nil {
return err
}
defer src.Close()
dst, err := sc.Create(remotePath)
if err != nil {
return fmt.Errorf("create remote %s: %w", remotePath, err)
}
defer dst.Close()
_, err = io.Copy(dst, src)
return err
}
// downloadFile copies a remote file to localPath over SFTP.
func downloadFile(client *ssh.Client, remotePath, localPath string) error {
sc, err := sftp.NewClient(client)
if err != nil {
return err
}
defer sc.Close()
src, err := sc.Open(remotePath)
if err != nil {
return err
}
defer src.Close()
dst, err := os.Create(localPath)
if err != nil {
return err
}
defer dst.Close()
_, err = io.Copy(dst, src)
return err
}sftp.Create creates or truncates. If the remote directory doesn't exist, the call fails. You need to sc.MkdirAll(path, 0755) first, or the error message will say "no such file or directory" pointing at the file, not the missing directory, and you'll waste time looking in the wrong place.
The SFTP client also exposes Stat, ReadDir, Mkdir, Remove, and Chmod, giving you a full filesystem API on any SSH host.
Jump Hosts and Bastion Proxying
Back to SSH. Production servers don't have public IPs. You connect to a bastion, then reach internal hosts through it. The result is still an *ssh.Client — once you have it, you use it exactly like any other: run commands, open SFTP, start sessions. The only difference is how you got there.
Before, a direct connection was one line:
client, err := ssh.Dial("tcp", host+":22", config)Through a bastion it's three steps: dial the bastion, tunnel a TCP connection to the target through it, then run the SSH handshake over that tunnel.
// dialViaBastion connects to targetAddr by tunneling through bastionAddr.
// Returns the target client, a cleanup func that closes all three underlying
// resources in the correct order, and any error. Call cleanup() when done.
func dialViaBastion(bastionAddr, targetAddr, user, keyPath string) (*ssh.Client, func(), error) {
config, err := newSSHConfig(user, keyPath)
if err != nil {
return nil, nil, err
}
bastion, err := ssh.Dial("tcp", bastionAddr+":22", config)
if err != nil {
return nil, nil, fmt.Errorf("dial bastion: %w", err)
}
ok := false
defer func() {
if !ok {
bastion.Close()
}
}()
conn, err := bastion.Dial("tcp", targetAddr+":22")
if err != nil {
return nil, nil, fmt.Errorf("dial target via bastion: %w", err)
}
defer func() {
if !ok {
conn.Close()
}
}()
ncc, chans, reqs, err := ssh.NewClientConn(conn, targetAddr+":22", config)
if err != nil {
return nil, nil, fmt.Errorf("handshake to target: %w", err)
}
target := ssh.NewClient(ncc, chans, reqs)
cleanup := func() {
target.Close()
conn.Close()
bastion.Close()
}
ok = true
return target, cleanup, nil
}The function returns three values: the client, a cleanup function, and an error. On the success path, the caller calls cleanup() when done and all three resources close in the right order. On any error path, ok stays false and the defers handle it.
Usage:
target, cleanup, err := dialViaBastion(bastionAddr, targetAddr, user, keyPath)
if err != nil {
log.Fatal(err)
}
defer cleanup()
// use target normally: run commands, open SFTP, etc.
out, err := runCommand(target, "hostname")defer cleanup() is one line. The caller doesn't need to know about conn or bastion at all.
SSH Remote Runner
A tool that connects to a remote host via SSH key auth, runs system commands (hostname, disk usage, uptime), and uploads a configuration file via SFTP. All in pure Go without shelling out to the ssh binary.
Input: Environment variables SSH_HOST, SSH_USER, and SSH_KEY pointing to the target server and private key
Output: Command output streamed to stdout, confirmation of file upload to /tmp/app.conf
Full source: examples/ssh-remote-runner
Key Takeaways
x/crypto/sshgives you a full SSH client in pure Go. No subprocess, nosshbinary dependency, no mixed stdout/stderr blobs- The
TimeoutonClientConfigonly covers the TCP dial. Use a context with a deadline to protect against hung handshakes and unresponsive commands - One
ssh.Clientsupports many sessions over a single connection. EachSessionis single-use: callRunorCombinedOutputonce, then open a new session for the next command - Goroutines make fleet operations trivial, but cap concurrent connections with a semaphore. Without a cap, large fleets will exhaust open file descriptors
sftp.Createfails if the remote directory doesn't exist. CallMkdirAllfirst or the error message will point at the file and you'll look in the wrong place- Never use
InsecureIgnoreHostKeyin production. Useknownhosts.Newso host key changes are caught, not silently accepted - Bastion proxying uses
bastionClient.Dial("tcp", targetAddr)to tunnel through. Close the target client before the bastion client or you'll get broken pipe errors on cleanup
🎁 devctl ssh can run commands and push files to remote machines. What if devctl container could pull images, start containers, and stream logs without shelling out to docker at all?