01 - What is Protobuf

📋 Jump to Takeaways

Protocol Buffers (protobuf) is a binary serialization format created by Google. You define your data structure in a .proto file, run a compiler, and get generated code in your language of choice. The generated code handles encoding and decoding for you.

The Problem with JSON

JSON is human-readable and universal. Every language can parse it. But it has real costs.

{
  "id": 1,
  "url": "https://example.com",
  "short_code": "abc123",
  "clicks": 42,
  "created_at": "2026-04-17T10:00:00Z"
}

That's 120+ bytes for five fields. Every field name is repeated in every message. Numbers are stored as text. There's no schema, so the sender can put anything in there and the receiver won't know until it tries to parse it.

In a system where services exchange millions of messages per second, those extra bytes and that parsing time add up.

How Protobuf Works

You write a .proto file that describes your data:

syntax = "proto3";

message Link {
  int64 id = 1;
  string url = 2;
  string short_code = 3;
  int64 clicks = 4;
  string created_at = 5;
}

Each field has a name, a type, and a number. The number is what actually gets sent over the wire, not the name. Field 1 is id, field 2 is url, and so on. This is why protobuf messages are so small.

You run the protobuf compiler (protoc) and it generates Go structs with marshal/unmarshal methods. You never write serialization code by hand.

Binary vs Text

JSON encodes the number 42 as two ASCII characters: 4 and 2. Protobuf encodes it as a single byte using variable-length encoding (varint). Small numbers take fewer bytes.

Strings are stored as a length prefix followed by raw bytes. No quotes, no escaping, no delimiters.

The result: protobuf messages are typically 3 to 10 times smaller than the equivalent JSON. Parsing is also faster because there's no tokenizer, no string matching, no type coercion.

Schema as a Contract

A .proto file is a contract. Both the sender and receiver agree on the shape of the data. If someone adds a field, removes a field, or changes a type, the compiler catches it. You don't find out at runtime with a json: cannot unmarshal string into Go value of type int error.

This matters when multiple teams own different services. The .proto file lives in a shared repo, and everyone generates code from it. One source of truth.

When to Use Protobuf

Use it for service-to-service communication. Internal APIs, microservices, anything where both sides are machines. It's great when you care about payload size (mobile, IoT), throughput (high-frequency trading, telemetry), or contract enforcement (multiple teams, multiple languages).

Don't use it for public APIs that developers debug with curl. Don't use it for config files. Don't use it when JSON is fine and you're optimizing for nothing.

Installing protoc

You need the protobuf compiler and the Go plugins. On macOS:

brew install protobuf
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Make sure $GOPATH/bin is in your PATH:

export PATH="$PATH:$(go env GOPATH)/bin"

Verify:

protoc --version
# libprotoc 28.x

Shortener Project Setup

We'll build a link shortener throughout this course. Start the project:

mkdir shortener && cd shortener
go mod init shortener
mkdir proto

Create proto/link.proto:

syntax = "proto3";

package shortener;

option go_package = "shortener/pb";

message Link {
  int64 id = 1;
  string url = 2;
  string short_code = 3;
  int64 clicks = 4;
}

We'll generate Go code from this in the next lesson. For now, notice the structure: syntax declares the version, package is the protobuf namespace, go_package tells the Go plugin where to put the generated code.

Key Takeaways

Protobuf is a binary serialization format, smaller and faster than JSON
You define data in .proto files and generate code with protoc
Field numbers (not names) are sent over the wire, which keeps messages compact
The .proto file acts as a contract between services
Use protobuf for service-to-service communication, not for public REST APIs
Install protoc, protoc-gen-go, and protoc-gen-go-grpc to get started