collapser

A gRPC sidecar that collapses duplicate in-flight requests into one backend call.

I wrote this to learn how sidecars, Kubernetes and Istio actually behave rather than reading about them. This page is what I measured, including the parts I got wrong on the first pass.

Source Raw measurements

What it does

When several callers ask a service the same question at the same time, the service answers it once per caller. The sidecar sits in front and gives them all the same answer from a single backend call.

The first request for a given key calls the backend. Anything arriving while that call is in flight waits on the same result instead of starting its own. A short result cache (100 ms) covers the gap just after a call finishes.

Envoy already does load balancing, retries and telemetry for this traffic, but it has no concept of two requests being the same request. That gap is the whole reason this exists.

Benchmarks

Three separate paths, benchmarked separately. Mixing them in one benchmark is the mistake I made first — see below.

Path	Time	Bytes	Allocs
Cache hit	45.7 ns	0 B	0
Joining an in-flight call	backend-bound	54 B	0
Leading a call	1,594 ns	552 B	10
No sidecar (baseline)	0.33 ns	0 B	0

Go 1.25.5, 11th Gen Core i7-1165G7, 8 threads. For the join path the number that means anything is allocations, not time — a waiter's wall clock is just however long the backend takes. Zero allocations there is the result of waiters sharing one channel instead of each registering their own.

Under load, in process

Test	Requests	Backend calls	Errors
10,000 concurrent, one key	10,000	2	0
60 s sustained, 100 workers	391,400	392	0

Race detector on. Goroutine count and heap were flat across 100,000 requests.

On a cluster

Deployed to a local Kubernetes cluster (kind) with Istio 1.28.1, Envoy sidecars injected and mTLS set to STRICT. Backend calls counted by the backend itself, not by the sidecar reporting on its own work.

Concurrent requests	Straight to backend	Through sidecar
100	100 calls	1 call
500	500 calls	1 call
1,000	1,000 calls	1 call
2,000	—	1 call

For the 2,000 run I checked three sources that don't depend on each other: the backend's own counter said 1, Envoy's istio_requests_total said 1, and the sidecar's metrics said 1.

What Istio saw

A load generator running inside the mesh, sending 600 concurrent requests spread over 25 distinct payloads every 250 ms. Both hops are reported by Envoy, so these are the mesh's numbers rather than mine.

Istio mesh dashboard in Grafana showing 1.42K requests per second total, with collapser-proxy receiving 1.36K req/s and hello-backend receiving 56.38 req/s, both at 100% success. — Istio mesh dashboard. The sidecar takes 1.36K req/s; the backend behind it sees 56 req/s. Same traffic, both measured by Envoy.

Kiali traffic graph showing loadgen sending traffic to collapser-proxy, which sends traffic to hello-backend, all edges healthy at 100 percent success. — Kiali, same window. The lock icons are Istio's mTLS between workloads — the sidecar itself speaks plaintext and never knows about it.

Over that run: 584,000 requests reached the sidecar and 24,304 reached the backend. The ratio is capped by how many distinct payloads the generator uses, not by the sidecar — with one payload it collapses to a single call.

What I got wrong

These were all in the version I had already written up before I went back and measured properly.

My headline benchmark measured the wrong thing. The "high contention" benchmark used a backend slower than the cache TTL, so nearly every iteration was a cache hit rather than an actual collapse. It reported 61 ns/op; the dedicated cache benchmark reported 58. Same code path, and I had been quoting it as the collapse number.
The collapse path was not allocation-free, so I made it one. Each waiter allocated its own channel and got appended to a slice. Measured honestly that was 3 allocations per request. Replacing it with a single channel the leader closes brought it to zero and deleted about 45 lines, including the locking that the channel now handles.
The repo did not compile from a fresh clone. Generated protobuf files were gitignored and nothing regenerated them. Anyone cloning it got a build error on the first command.
Request metadata was silently dropped. Because the backend call runs on a detached context, incoming headers never made it through on unary calls — while the streaming path forwarded them correctly. Inside a mesh that quietly breaks trace propagation.
The default could serve one caller's data to another. Two callers sending identical payloads shared a response even when authenticated as different users. Headers the backend varies on now have to be named in COLLAPSER_KEY_HEADERS so those requests stay separate. This one bothers me most, because the code looked correct.

Limitations

Server-streaming calls are treated as unary. Without a service descriptor the proxy can't tell them apart on the incoming connection, so only the first response frame comes back. Client-streaming and bidi are detected and skip collapsing entirely.
Each replica collapses on its own, so N replicas can still make N backend calls for the same key. Sharing that state across replicas would cost the latency the whole thing exists to save.
The passthrough codec registers itself globally under the name proto. That is what makes the proxy work without generated stubs, but it means the package can't be imported into a process that needs normal protobuf marshalling.
Everything here was measured on one laptop and a local cluster. I have not run it against real production traffic.