1 part · ongoing
Building an LLM Gateway
An OpenAI-compatible gateway in Go, built one phase at a time: streaming, two providers, API keys, distributed rate limits, fallback and metrics.
Start from Part 1I'm a software engineer with 3.4 years of experience, interested in backend engineering. The posts here cover what I've learned and what I'm curious about.
Series
All series2 parts · ongoing
Distributed Systems
How data stays correct when it lives on many machines. One topic per part, each with step-through animations.
Start from Part 18 parts · complete
Understanding Kubernetes
From what a cluster is to GitOps with Argo CD. Hands-on, one concept per part.
Start from Part 1Latest
See all posts- Oct 24 LLM Gateway, Part 1: The goroutine leak hiding in every naive streaming proxyA streaming proxy in Go that cleans up when the client leaves. One context for the whole request, an idle timer for a silent provider, and pprof to prove that 1,000 canceled streams leave nothing behind.
- Oct 10 Distributed Systems, Part 2: Replication by handBuilding single-leader replication with Express, Redis and Nginx, one step at a time: shared state, replication offsets, read-your-writes, resync, and a manual failover that loses an acknowledged write.
- Oct 09 Distributed Systems, Part 1: ReplicationKeeping copies of the same data on several nodes. Leaders and followers, sync and async, failover, replication lag, multi-leader and leaderless designs, and quorums, each with a step-through animation.
- Sep 12 Understanding Kubernetes — Part 8Argo CD: what it is, installing it on DigitalOcean with the 1-Click app, and wiring up a pipeline where pushing code triggers GitHub Actions, which builds an image and commits the new tag to a deployment repo that Argo CD syncs to the cluster.
- Aug 26 Understanding Kubernetes — Part 7Resource requests and limits, what a CPU number actually buys you, the metrics-server, and handing the replica count over to a Horizontal Pod Autoscaler.