LLM Gateway, Part 1: The goroutine leak hiding in every naive streaming proxy

I am building an LLM gateway in Go. It is one OpenAI-compatible API in front of several model providers. This series follows the build one phase at a time. The code is on GitHub.

Phase 1 is a streaming proxy. It accepts a chat request, swaps in the real provider key and streams the answer back token by token. A proxy like this is easy to write in 30 lines. It is also easy to write one that leaks a goroutine every time a user closes a browser tab.

This post shows the leak, the fix and the proof. Every number in it comes from a test that I ran.

The diagrams play one step at a time. Next plays the next step, Prev goes back one step, and Play all plays the remaining steps. The tabs above a diagram change the scenario.

1. The streaming loop

A provider streams its answer as server-sent events. Each event is one line that starts with data:, and a blank line follows it. The gateway must send each event to the client as soon as it arrives. If the gateway buffers the events, the user sees nothing until the whole answer is ready.

The core of the proxy is a loop that reads a chunk from the provider, writes it to the client and flushes it.

buf := make([]byte, 32*1024)
rc := http.NewResponseController(w)

for {
	n, err := res.Body.Read(buf)
	if n > 0 {
		w.Write(buf[:n])
		rc.Flush() // send the chunk now, do not wait for a full buffer
	}
	if err != nil {
		return
	}
}

This loop works while the client and the provider both behave. The problems start when one of them stops.

2. The leak: a client that leaves

Users leave all the time. They close the tab, stop the request or lose the network. A model can also think for many seconds before it sends the first token. During that time, the proxy waits in client.Do for the response headers.

Select Naive proxy and step through what happens when the user leaves during that wait.

The naive proxy fails in two ways:

• It does not notice. A proxy that ignores req.Context() finds a closed client connection only when a write fails. While the provider is silent, the proxy writes nothing, so it never finds out.
• It cannot stop the upstream request. The request uses context.Background(), either directly or through http.NewRequest. Nothing can cancel it, and http.DefaultClient has no timeout.

So the goroutine waits for as long as the provider takes. When the first token arrives, the proxy writes it, the write fails and the proxy stops. If the provider never answers, the goroutine waits forever. During the wait, the provider works on a request that nobody will read, and it charges for that work.

A proxy that also ignores write errors is worse. It reads the whole answer, and the provider charges for every token.

Measured

I tested this with a naive proxy that checks every write error. The mock provider waited 10 minutes before its first byte. I sent 100 streaming requests, and each client disconnected after 1 second.

Each canceled request left 3 goroutines in the naive proxy. The goroutine profile showed all three:

• the handler, waiting in client.Do
• persistConn.readLoop, which reads from the upstream connection
• persistConn.writeLoop, which writes to the upstream connection

The mock provider logged 0 cancels. As far as it knew, 100 clients still waited for an answer.

3. The fix: one context for the whole request

net/http already knows when the client leaves. The documentation of Request.Context says it: for incoming server requests, the context is canceled when the client’s connection closes.

So the fix is to make the upstream request a child of that context:

ctx, cancel := context.WithCancelCause(req.Context())
defer cancel(nil)

upReq, err := http.NewRequestWithContext(ctx, req.Method, upstreamURL, bytes.NewReader(body))

Select With the client’s context in the diagram above. The cancel now goes through five links:

1. The client closes its TCP connection.
2. The gateway’s server sees it and cancels req.Context().
3. ctx is a child of req.Context(), so it is canceled too.
4. The HTTP client sees the canceled ctx. Do or Read returns at once, and the Transport closes the upstream connection.
5. The provider’s server sees the closed connection and cancels its own request context.

Every link must work. If one link is missing, the chain stops there. For example, a provider that waits with time.Sleep instead of select on ctx.Done() continues after link 5. My first mock had exactly this bug.

With the fix, the same test leaves the gateway at 7 goroutines, and the mock logs 100 cancels.

WithCancelCause instead of WithCancel is for the next section. A cancel can have more than one reason, and the gateway must know which reason it was.

4. The silent provider

The client context protects against a client that leaves. It does not protect against a provider that stops sending while the client still waits. The TCP connection stays open, no bytes arrive, and Read blocks.

The usual timeouts do not fit a stream:

A stream needs a limit on silence, not on total time. The gateway uses an idle timer that resets each time a chunk arrives:

idle := time.AfterFunc(g.idleTimeout, func() { cancel(errIdleTimeout) })
defer idle.Stop()

for {
	n, err := res.Body.Read(buf)
	if n > 0 {
		idle.Reset(g.idleTimeout) // the provider is alive: start the 60 s again
		// write and flush ...
	}
	// handle err ...
}

The timer runs from the start of the request, so it also covers the wait for the first byte. One timer covers both cases. Before the first byte, the gateway returns 504 Gateway Timeout. During the stream, the status line is already sent, so the gateway can only end the stream.

5. Each exit has its own cause

A canceled context only says that the request stopped. context.Cause(ctx) says why. The gateway uses the cause to choose the log level and the response:

A client disconnect is INFO, not ERROR. The user chose to leave, and nothing is broken. If disconnects were errors, the alerts would fire all day and people would learn to ignore them.

6. How to prove that there is no leak

A test that passes does not prove that there is no leak. The goroutine count does. Go’s net/http/pprof package shows the count and the stack of every goroutine in a running program.

The package registers its handlers when you import it. The _ import runs its init function, which adds the handlers to http.DefaultServeMux:

import _ "net/http/pprof"

go func() {
	if err := http.ListenAndServe("localhost:6060", nil); err != nil {
		slog.Error("pprof server failed", "err", err)
	}
}()

Two details are important:

• The pprof server listens on localhost, not on all interfaces. pprof shows stack traces and memory, and a CPU profile costs CPU. Never put it on a public port.
• The gateway itself uses its own ServeMux, so the pprof routes do not exist on the public port. A request there gets 404.

The baseline

curl -s 'localhost:6060/debug/pprof/goroutine?debug=1' | head -1
# goroutine profile: total 7

An idle gateway has 7 goroutines. Five always run, and two exist only because the profile request is running:

You cannot see 5 through HTTP. The tool always counts itself.

7. 7 → 9 is not a leak

After one normal stream, the count goes to 9 and stays there. This looks like a leak. It is the connection pool.

Body.Close does one of two things:

• The body was read to io.EOF. The Transport keeps the connection in its idle pool for the next request to the same host. For a provider on HTTPS, that saves a TCP and TLS handshake on every request.
• The body was not read to the end. The Transport cannot use the connection again, so it closes it.

HTTP/1.1 sends one response after the other on the same byte stream, with no request ID. If the rest of the old body stays on the connection, the next request would read those bytes as the start of its own response. To use the connection again, the Transport would have to read and discard the rest of the stream, which can be very long. So it closes the connection.

This is true for HTTP/1.1, which the gateway uses with the mock. Over HTTPS, Go usually uses HTTP/2 instead. HTTP/2 sends many streams on one connection, each with its own ID. A canceled request resets only its own stream, and the connection stays open.

A pooled connection keeps its readLoop and writeLoop goroutines. The pool has two limits, and they keep this growth small:

I measured both limits:

So a leak is not “the count went up”. A leak is a count that grows with each request and never comes down.

8. The 1,000-stream test

The plan for this phase has one check for leaks: after 1,000 canceled streams, the goroutine count must return to the baseline.

I used the mock provider in a mode that stalls for 10 seconds after the first word. curl -m 0.5 disconnected each client during the stall, and xargs -P 50 ran 50 at a time:

seq 1000 | xargs -P 50 -I{} curl -sN -m 0.5 -o /dev/null \
  localhost:8080/v1/chat/completions \
  -d '{"model":"mock-stall","stream":true,"messages":[]}'

I read the count once per second while the test ran.

A leak test needs three readings: before, during and after. The reading during the load is important. It proves that the tool sees the load. If the count did not go up, the test would prove nothing.

The logs agreed with the chart. The gateway logged 1,000 client disconnected lines, and the mock logged 1,000 stream wait stopped: context canceled lines.

If there were a leak

Leaked goroutines usually share one stack. In ?debug=1 output, they appear as one group with a large count, for example 1000 @ .... Read that group from the top down. The top line shows what the goroutines wait on, such as chanrecv or selectgo. The first line that starts with main. gives the file and line number in your code.

9. Two traps

A wrapped ResponseWriter hides Flush

The gateway logs one line per request, with the status code. To get the status, middleware wraps the http.ResponseWriter in a struct that records it. After I added the middleware, every stream stopped after the first chunk. rc.Flush() returned feature not supported.

http.ResponseController looks for a Flush method on the writer. The wrapper embeds the http.ResponseWriter interface, and embedding promotes only the methods of that interface. Flush is not one of them. The fix is an Unwrap method, which ResponseController uses to find the original writer:

func (rs *statusRecorder) Unwrap() http.ResponseWriter {
	return rs.ResponseWriter
}

A server timeout that cuts streams

http.Server has several timeouts, and only some of them are safe for a streaming server:

The gateway sets the first three and never sets WriteTimeout.

10. What comes next

Phase 1 is done. All three checks of the plan pass:

• The official OpenAI Python SDK streams through the gateway. The only change is its base URL.
• A client that disconnects in the middle of a stream cancels the provider’s request, and the mock’s log shows it.
• After 1,000 canceled streams, the goroutine count is the same as before the test.

Phase 2 adds Anthropic behind the same API. The gateway must translate requests and streaming events between two different formats.

Check your understanding