Courses › System Design › Key Technologies

gRPC & Protobuf

Internal service-to-service RPC: Protobuf contracts, streaming, HTTP/2 load balancing, deadlines, retries and schema evolution.

gRPC is the default answer when an interviewer asks "how do your internal services talk to each other?" It's an RPC framework from Google that runs on HTTP/2 and encodes messages with Protocol Buffers (Protobuf), a compact binary format with a typed schema. You define a service once in a .proto file and generate client and server code in every language your teams use. Keep REST for public APIs, as covered in API Design. Use gRPC for internal service-to-service calls.

Why gRPC for internal calls

  • Typed contracts. The .proto file is the API. Generated stubs catch mismatched fields at compile time instead of in production.
  • Small, fast payloads. Protobuf messages are typically 3–10x smaller than the same JSON and much cheaper to parse, which adds up when one user request fans out to dozens of internal calls.
  • HTTP/2 multiplexing. Many concurrent calls share one TCP connection, with no head-of-line blocking at the HTTP layer and no per-request handshake.
  • Streaming built in. Server, client and bidirectional streams are part of the protocol, not bolted on.
  • Deadlines, cancellation and metadata. Timeouts propagate across hops, and headers (auth tokens, trace IDs) travel as metadata.

Defining a service

syntax = "proto3";

service OrderService {
  rpc GetOrder(GetOrderRequest) returns (Order);                    // unary
  rpc WatchOrder(WatchOrderRequest) returns (stream OrderEvent);    // server streaming
  rpc UploadItems(stream Item) returns (UploadSummary);             // client streaming
  rpc Chat(stream ChatMessage) returns (stream ChatMessage);        // bidirectional
}

message Order {
  string order_id    = 1;
  int64  user_id     = 2;
  repeated Item items = 3;
  OrderStatus status = 4;
}

The numbers (= 1, = 2) are field tags. They are what goes on the wire, not the field names, which is why Protobuf is compact and why schema changes need care (see below).

The four call types

Most systems are 95% unary calls. Mention streaming when the problem genuinely needs it, for example a gateway that receives SSE from users and a gRPC server stream from the inference service, as in LLM Inference.

Load balancing: the HTTP/2 gotcha

This is the most common gRPC follow-up question. HTTP/2 connections are long-lived and multiplexed: a client opens one connection and sends every request over it. An L4 load balancer balances *connections*, not requests, so it pins all of that client's traffic to a single backend. Add new pods during a traffic spike and they sit idle.

[Order service] -> [L4 LB] -> [Pricing pod 1: 100% of traffic]
[L4 LB] -> [Pricing pod 2: idle]
[L4 LB] -> [Pricing pod 3: idle]

Three fixes, in order of how often you'll see them:

  • L7 proxy or service mesh. An Envoy sidecar or proxy understands HTTP/2 and balances each *request* across backends. This is the standard answer in Kubernetes. See Nginx & Envoy.
  • Client-side load balancing. The gRPC client resolves all backend addresses (via DNS or a service registry), keeps a connection to each, and round-robins requests itself. No extra hop, but every client needs the logic and fresh endpoint lists.
  • Lookaside balancer. Clients ask a central service which backend to use (gRPC's xDS support works this way). Common at large scale, rarely needed in an interview.

[!WARNING] Kubernetes Services are L4 A plain Kubernetes ClusterIP Service balances connections, so gRPC traffic ends up stuck on whichever pod each client first connected to. Use a headless Service with client-side balancing, or a mesh like Istio or Linkerd. See Kubernetes.

Deadlines and cancellation

Every gRPC call should carry a deadline: an absolute time by which the caller needs an answer. gRPC sends it to the next hop, which passes the *remaining* time to its own downstream calls. If the user's request has 300 ms left when it reaches the pricing service, pricing doesn't start a fresh 2-second timeout.

When the deadline passes or the caller gives up, cancellation propagates down the chain, so servers stop doing work nobody will read. This is the built-in version of the deadline advice in Resilience Patterns.

[!TIP] Say the number "The gateway sets an 800 ms deadline. Each gRPC hop inherits what's left, and downstream services drop work once it expires" is a stronger answer than "we add timeouts."

Errors and retries

gRPC has its own status codes instead of HTTP ones. The ones worth knowing:

gRPC supports retry policies configured per method, with limits and backoff. Only retry calls that are safe to repeat. A DEADLINE_EXCEEDED on ChargeCard doesn't tell you whether the charge happened, so the call needs an idempotency key. See Safe Retries & Idempotency.

Schema evolution

Services deploy independently, so old and new versions of a message are always in flight at the same time. Protobuf handles this well if you follow the rules:

  • Add fields freely with new tag numbers. Old readers skip unknown fields, and new readers see default values (0, "", empty list) when a field is missing.
  • Never change a field's tag number or type. The tag is the identity on the wire.
  • Never reuse a deleted field's tag. Mark it reserved so nobody can, otherwise an old client's data will be misread as the new field.
  • Treat every field as optional. In proto3 a missing field and a zero value look the same unless you use optional or wrapper types. Don't put "this field must be set" logic into the schema.
  • Make breaking changes with a new method or a new package version (orders.v2.OrderService) and run both until callers migrate.
message Order {
  reserved 5;              // was: string coupon_code
  reserved "coupon_code";
  string order_id = 1;
  ...
}

The same rules apply when Protobuf is used for events on Kafka or a pub/sub bus, usually with a schema registry enforcing compatibility. See Zero-Downtime Migrations.

[!NOTE] Lint in CI Tools like buf check every .proto change against the previous version and fail the build on a breaking change. That turns "please be careful" into an enforced rule.

Browsers and public APIs

Browsers can't make raw gRPC calls, because JavaScript has no access to HTTP/2 framing and trailers. The options:

  • REST or GraphQL at the edge, gRPC inside. The most common setup. An API Gateway or BFF translates public JSON requests into internal gRPC calls.
  • gRPC-Web. A browser-compatible variant that needs a proxy (Envoy) to translate. Works, but supports only unary and server streaming.
  • Transcoding. Annotate the .proto with HTTP routes and let a gateway (Envoy, Google Cloud Endpoints, grpc-gateway) expose the same service as JSON/REST automatically.
[Browser / Mobile] -> [API Gateway: REST + JSON] -> [Order service]
[Order service] -> [Pricing service: gRPC]
[Order service] -> [Inventory service: gRPC]
[Order service] -> (Kafka: Protobuf events)

When not to use gRPC

  • Public or third-party APIs. Partners expect REST and JSON, readable docs and curl.
  • Simple systems with few services. The codegen toolchain and proxy setup aren't worth it for three services that talk rarely.
  • Debugging by hand. Binary payloads need tools like grpcurl instead of a browser or curl. Small cost, but real.
  • Async work. If the caller doesn't need an answer right now, use a message queue or pub/sub instead of any synchronous RPC.

Alternatives you might name

[!INTERVIEW] Checkout service calling pricing and inventory "Checkout calls Pricing and Inventory over gRPC with Protobuf contracts owned by each team and checked by buf in CI. The gateway sets an 800 ms deadline that propagates through each hop. Traffic goes through Envoy sidecars, so requests are balanced per call rather than per HTTP/2 connection, and the sidecars retry UNAVAILABLE once with jitter. ReserveStock takes an idempotency key, so a retry after a timeout can't reserve twice. Public clients never see gRPC: the API Gateway exposes REST and translates."