04 - Serialization

Protobuf's whole point is turning structs into bytes and back. This lesson covers how marshaling and unmarshaling work, what the binary format looks like, and how it compares to JSON in practice.

Marshal and Unmarshal

The proto package handles serialization:

import "google.golang.org/protobuf/proto"

link := &pb.Link{
    Id:        1,
    Url:       "https://example.com",
    ShortCode: "abc123",
    Clicks:    42,
}

// struct -> bytes
data, err := proto.Marshal(link)
if err != nil {
    log.Fatal(err)
}
fmt.Println(len(data)) // ~35 bytes

// bytes -> struct
var decoded pb.Link
err = proto.Unmarshal(data, &decoded)
if err != nil {
    log.Fatal(err)
}
fmt.Println(decoded.GetUrl()) // "https://example.com"

That's it. No field tags to configure, no custom marshalers. The generated code knows how to encode and decode itself.

Size Comparison

Let's compare the same data in JSON vs protobuf:

import "encoding/json"

link := &pb.Link{
    Id:        1,
    Url:       "https://example.com",
    ShortCode: "abc123",
    Clicks:    42,
}

// Protobuf
pbData, _ := proto.Marshal(link)
fmt.Printf("protobuf: %d bytes\n", len(pbData))

// JSON
jsonData, _ := json.Marshal(link)
fmt.Printf("json:     %d bytes\n", len(jsonData))

Output:

protobuf: 35 bytes
json:     74 bytes

Protobuf is roughly half the size here. The difference grows with more fields and larger messages. In a system sending millions of messages, that's real bandwidth savings.

How the Wire Format Works

You don't need to memorize this, but understanding the basics helps when debugging.

Each field is encoded as a key-value pair. The key contains the field number and wire type (varint, length-delimited, etc.). The value follows immediately.

For our Link message:

• Field 1 (id = 1): encoded as varint, takes 1 byte for the key + 1 byte for the value = 2 bytes
• Field 2 (url): encoded as length-delimited, key + length + raw string bytes
• Field 3 (short_code): same as url
• Field 4 (clicks = 42): varint, 2 bytes total

Field names are never sent. Only field numbers. This is why protobuf is compact and why field numbers must never change once assigned.

Zero Values Are Not Serialized

Proto3 skips fields that have their default zero value:

link := &pb.Link{
    Id: 0,  // won't be serialized
    Url: "",  // won't be serialized
}

data, _ := proto.Marshal(link)
fmt.Println(len(data)) // 0 bytes

An empty message serializes to zero bytes. This is efficient but means you can't tell the difference between "field was explicitly set to zero" and "field was not set." If that distinction matters, use the optional keyword in your proto file.

JSON Encoding for Debugging

Sometimes you need to see protobuf messages as JSON, for logging or debugging. The protojson package handles this:

import "google.golang.org/protobuf/encoding/protojson"

link := &pb.Link{
    Id:        1,
    Url:       "https://example.com",
    ShortCode: "abc123",
}

jsonBytes, _ := protojson.Marshal(link)
fmt.Println(string(jsonBytes))
// {"id":"1","url":"https://example.com","shortCode":"abc123"}

Notice that protojson uses camelCase field names by default (protobuf convention) and encodes int64 as strings (because JavaScript can't handle 64-bit integers). This is the canonical JSON mapping defined by the protobuf spec.

To unmarshal JSON back into a protobuf message:

var link pb.Link
err := protojson.Unmarshal(jsonBytes, &link)

Proto Text Format

There's also a text format, useful for config files and test fixtures:

import "google.golang.org/protobuf/encoding/prototext"

text := prototext.Format(link)
fmt.Println(text)
// id: 1
// url: "https://example.com"
// short_code: "abc123"

You'll rarely use this in production, but it's handy for debugging and writing test data.

Key Takeaways

• proto.Marshal converts a struct to bytes, proto.Unmarshal converts bytes back
• Protobuf messages are typically 2 to 10 times smaller than JSON
• Field numbers (not names) are sent over the wire
• Zero-value fields are not serialized in proto3
• Use protojson for JSON encoding/decoding of protobuf messages
• protojson uses camelCase and encodes int64 as strings (JavaScript compatibility)
• prototext gives a human-readable text format for debugging