02 - File I/O Deep

🎁 os.ReadFile is fine until the file is bigger than your RAM. Real systems tools read a few kilobytes at a time and jump around inside a file without ever loading the whole thing. How do they do it?

In Lesson 1 you saw that an *os.File wraps a file descriptor. Now we use that descriptor for real: opening files with precise control, reading and writing raw bytes, jumping to any offset, inspecting metadata, following symlinks, and confining I/O to a directory so untrusted input can't escape it.

Opening Files with OpenFile

os.Open and os.Create are convenience shortcuts. The real workhorse is os.OpenFile, which takes explicit flags and a permission mode:

// os.Open(name) is really:
f, err := os.OpenFile("data.txt", os.O_RDONLY, 0)

// os.Create(name) is really:
f, err := os.OpenFile("data.txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)

You combine the flags with |. The ones that matter:

• O_RDONLY, O_WRONLY, O_RDWR: the access mode (pick one)
• O_CREATE: create the file if it doesn't exist
• O_EXCL: with O_CREATE, fail if it already exists
• O_APPEND: every write goes to the end of the file
• O_TRUNC: empty the file on open

An append-only log file is a common combination:

f, err := os.OpenFile("app.log", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
if err != nil {
    log.Fatal(err)
}
defer f.Close()
f.WriteString("service started\n")

The 0644 is the permission mode, applied only when the file is created. The three digits after the leading 0 set read/write/execute for the file's owner, its group, and everyone else, so 0644 means the owner reads and writes and everyone else only reads (Lesson 3 covers this and how the umask can trim it). Pair O_CREATE with O_EXCL when you want "create only if new," which is the basis of a simple lock file:

_, err := os.OpenFile("service.lock", os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0644)
if errors.Is(err, os.ErrExist) {
    fmt.Println("already running")   // already running
}

Reading and Writing Bytes

At the lowest level you read into a byte slice you allocate yourself:

buf := make([]byte, 4)
n, err := f.Read(buf)
fmt.Println(n, string(buf[:n]))   // 4 data

Read fills up to len(buf) bytes and returns how many it actually got. Always slice with buf[:n], because the last read is usually short. When there's nothing left, Read returns io.EOF:

for {
    n, err := f.Read(buf)
    // ... process buf[:n] ...
    if err == io.EOF {
        break
    }
    if err != nil {
        log.Fatal(err)
    }
}

Writing is the mirror image, returning the number of bytes written:

n, err := f.Write([]byte("hello"))
fmt.Println(n)   // 5

Most days you won't hand-roll these loops. For lines, use bufio.Scanner; for a whole small file, os.ReadFile. You already used those in Go Essentials → Build a File Scanner. This lesson is about the layer underneath them.

Seeking to Any Offset

Every open file has a current offset: the position of the next read or write. Seek moves that position without transferring any bytes:

f.Seek(2_000_000, io.SeekStart)   // jump to byte 2,000,000
buf := make([]byte, 10)
f.Read(buf)                        // reads bytes 2,000,000 through 2,000,009

The second argument, whence, says what the offset is measured from:

• io.SeekStart: from the beginning of the file
• io.SeekCurrent: from the current position
• io.SeekEnd: from the end (use a negative offset)

Reading the last 128 bytes of a file, without touching the rest, is just a seek from the end:

f.Seek(-128, io.SeekEnd)
tail := make([]byte, 128)
n, _ := f.Read(tail)
fmt.Println(string(tail[:n]))

That's the answer to the opening question. A 40 GB file is no problem when you seek to the window you need and read a few kilobytes. You never load the whole thing.

File Metadata with Stat

os.Stat asks the kernel about a file without opening it for reading. It returns an os.FileInfo:

info, err := os.Stat("data.txt")
if err != nil {
    log.Fatal(err)
}
fmt.Println(info.Name())    // data.txt
fmt.Println(info.Size())    // 42
fmt.Println(info.Mode())    // -rw-r--r--
fmt.Println(info.IsDir())   // false

Stat is also the idiomatic way to test whether something exists, using the sentinel from Lesson 1:

if _, err := os.Stat("config.yaml"); errors.Is(err, os.ErrNotExist) {
    fmt.Println("no config, using defaults")
    // no config, using defaults
}

Symlinks and Lstat

A symbolic link is a small file that points at another path. You create and read the link target directly:

os.Symlink("data.txt", "latest")   // latest -> data.txt
target, _ := os.Readlink("latest")
fmt.Println(target)                 // data.txt

Here's the trap that catches people: os.Stat follows a symlink and reports on its target, while os.Lstat reports on the link itself. To ask "is this path a symlink?", you need Lstat:

info, _ := os.Lstat("latest")
fmt.Println(info.Mode()&fs.ModeSymlink != 0)   // true

os.Stat("latest") would instead describe data.txt. This difference matters any time you walk a directory tree and don't want to wander off through a link into another part of the filesystem.

Scoped I/O with os.Root

Suppose your program serves files from ./uploads and a user requests the path ../../etc/passwd. Naive path joining lets them escape the directory. Go 1.24 added os.Root to close this hole: every operation is confined to one directory, and any attempt to climb out fails.

root, err := os.OpenRoot("uploads")
if err != nil {
    log.Fatal(err)
}
defer root.Close()

f, err := root.Open("photo.jpg")   // fine: inside uploads
_ = f

_, err = root.Open("../../etc/passwd")
fmt.Println(err != nil)             // true: escape blocked

os.Root exposes the methods you'd expect, each scoped to the root: Open, Create, OpenFile, Mkdir, Stat, Remove, and more. Crucially it also blocks escapes that go through a symlink, which a manual filepath.Clean check quietly misses. Any time you open files from untrusted input, reach for os.Root instead of hand-rolled path validation.

Key Takeaways

• os.OpenFile(name, flag, perm) is the general open; os.Open and os.Create are shortcuts. Combine flags with |: one of O_RDONLY/O_WRONLY/O_RDWR, plus O_CREATE, O_EXCL, O_APPEND, O_TRUNC. perm applies only when the file is created.
• f.Read(buf) fills a byte slice and returns the count; slice results as buf[:n] and treat io.EOF as the end. f.Write returns bytes written. Use bufio.Scanner for lines, os.ReadFile for whole files.
• Every open file has a current offset. f.Seek(offset, whence) moves it, where whence is io.SeekStart, io.SeekCurrent, or io.SeekEnd. Seeking lets you read a small window of a huge file.
• os.Stat returns FileInfo (Name, Size, Mode, IsDir, ModTime) without opening the file; combine with errors.Is(err, os.ErrNotExist) to test existence.
• os.Symlink/os.Readlink manage links. os.Stat follows a link to its target; os.Lstat describes the link itself. Test info.Mode()&fs.ModeSymlink != 0 to detect one.
• os.Root (Go 1.24) confines all I/O to one directory and blocks ../ and symlink escapes. Use it for untrusted paths instead of manual validation.

🎁 You just created a file with mode 0644. But depending on how the process is running, the file that lands on disk might not have those permissions at all, and if your program is running as root, that's a security problem waiting to happen. Who really decides the final permissions, and how do you give up power you don't need?