Building an LLM Gateway

An OpenAI-compatible gateway in Go, built one phase at a time: streaming, two providers, API keys, distributed rate limits, fallback and metrics.

1 parts ongoing about 15 min in total

01

LLM Gateway, Part 1: The goroutine leak hiding in every naive streaming proxy

A streaming proxy in Go that cleans up when the client leaves. One context for the whole request, an idle timer for a silent provider, and pprof to prove that 1,000 canceled streams leave nothing behind.

13 min