Distributed Systems, Part 2: Replication by hand

Part 1 covered replication in theory. This part builds it on a laptop and then breaks it: three Express apps behind Nginx, a Redis leader, and a Redis follower.

The build has four steps. Each step starts as a copy of the previous one and adds one thing. For every experiment I wrote down a prediction first, ran the commands, and compared. Several predictions were wrong, and those are the most useful parts of this post.

The diagrams work like the ones in Part 1: the first step plays when a diagram scrolls into view, Next and Prev move one step, and Play all runs the rest.

The setup

I left out Redis Sentinel, Redis Cluster, Kubernetes and ORMs on purpose. Each of them would detect failures or move traffic for me, and the point was to see those problems directly.

The Dockerfile is the same in all four steps:

FROM oven/bun:1
WORKDIR /app

# Dependencies first, so a code change does not run the install again
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile

COPY index.ts tsconfig.json ./

EXPOSE 3000
CMD ["bun", "index.ts"]

Docker caches each layer. Copying package.json and bun.lock before the code means a change to index.ts reuses the cached bun install layer. --frozen-lockfile fails the build if the lockfile doesn’t match package.json, so every image gets the exact versions I tested.

1. Three instances with local state

The first version keeps a counter in memory.

import express from "express";
import os from "os";

const app = express();
const PORT = process.env.PORT || 3000;

let count = 0;

app.get("/count/local", (req, res) => {
  count = count + 1;
  res.json({
    instance: os.hostname(),
    count
  });
})

app.listen(PORT, () => {
  console.log(os.hostname() + " " + PORT)
})

count is a module-level variable. Bun runs the module once when the process starts, so there is one count per process, shared by every request that process handles. os.hostname() returns the container’s hostname, so each response says which instance answered.

Compose starts three copies of the same image, and Nginx sits in front:

services:
  app1:
    build: ./app
    hostname: app1
    environment:
      PORT: 3000

  app2:
    build: ./app
    hostname: app2
    environment:
      PORT: 3000

  app3:
    build: ./app
    hostname: app3
    environment:
      PORT: 3000

  nginx:
    image: nginx
    ports:
      - "8080:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app1
      - app2
      - app3

events {}
http {
  upstream app { server app1:3000; server app2:3000; server app3:3000; }
  server { listen 80; location / { proxy_pass http://app; } }
}

All three apps listen on port 3000 without a conflict. Each container has its own network namespace, with its own IP address and its own table of ports. Only Nginx publishes a port to the host: 8080 on the laptop goes to port 80 in the Nginx container.

upstream app is the list of backends. Nginx resolves app1, app2 and app3 through Docker’s DNS once, when it starts. That detail causes a problem in step 4. proxy_pass http://app sends each request to the next server in the list, in round-robin order.

The experiment

for i in $(seq 1 9); do curl -s localhost:8080/count/local; echo; done

{"instance":"app1","count":1}
{"instance":"app2","count":1}
{"instance":"app3","count":1}
{"instance":"app1","count":2}
{"instance":"app2","count":2}
{"instance":"app3","count":2}
{"instance":"app1","count":3}
{"instance":"app2","count":3}
{"instance":"app3","count":3}

Nine requests produced three counters stuck at 3. Each request increments the variable in whichever process handles it, and the other two processes never see the change. The answer depends on which instance Nginx picks.

An instance that keeps state in memory can’t be copied freely. To run many instances, the state has to live outside them.

2. Shared state in Redis

The change

Compose gets a Redis service, and each app gets the address of the leader:

services:
  app1:
    build: ./app
    hostname: app1
    environment:
      PORT: 3000
      REDIS_LEADER_HOST: redis

  # app2 and app3 get the same line

  redis:
    image: redis:7

The Redis service has no ports, because only the apps need it, and they reach it on the Docker network as redis:6379. It has no volume either. The leader’s address is an environment variable because it is configuration, not code: in step 4 I change this one value to fail over.

bun add ioredis adds the client. I picked ioredis over the official redis package because three constructor options turn off everything that hides a failure:

• enableOfflineQueue: false. With no connection, fail the command now instead of queueing it until the connection returns.
• maxRetriesPerRequest: 1. Retry a command once, not 20 times.
• commandTimeout: 1000. Fail if Redis doesn’t reply within one second.

With the defaults, a request to a stopped Redis can hang for about 70 seconds while ioredis reconnects with growing delays. That hides exactly what this lab is trying to see.

import express from "express";
import os from "os";
import Redis from "ioredis";

const app = express();
const PORT = process.env.PORT || 3000;

let count = 0;
const leader = new Redis({
  host: process.env.REDIS_LEADER_HOST ?? "redis",
  port: 6379,
  enableOfflineQueue: false,  // no connection? fail now, do not queue
  maxRetriesPerRequest: 1,     // retry a command one time only
  commandTimeout: 1000,        // no reply in 1 s? fail
});

app.get("/count/local", (req, res) => {
  count = count + 1;
  res.json({
    instance: os.hostname(),
    count
  });
})
leader.on("error", (err) => console.error("redis:", err.message));

app.get("/count", async (req, res) => {
  try {
    const count = await leader.incr("counter");
    res.json({
      instance: os.hostname(),
      count
    })
  } catch (e) {
    console.error("An error occurred: " + e)
    res.status(503).json({
      error: e,
      instance: os.hostname()
    })
  }
})

app.listen(PORT, () => {
  console.log(os.hostname() + " " + PORT)
})

Line by line:

• One client at module level. Like count in step 1, leader is created once per process and shared by all requests. It holds one TCP connection to Redis.
• The error listener. ioredis emits error on every failed reconnect. Without a listener, Node treats it as an unhandled error and logs a stack trace each time.
• INCR counter. Redis runs one command at a time. INCR reads, adds one and writes as a single step, so two apps can never read the same value and both write the same result. A GET followed by a SET from the app would lose updates.
• 503 on failure. My first version only logged in the catch block and never responded. Express then left the connection open, Nginx waited 60 seconds for a reply and returned 504. The fast-fail options were useless until the handler answered.

error: e has a bug I didn’t notice yet: JSON.stringify skips the fields of an Error, so the client receives {}. Step 3 fixes it.

One counter

Nine requests now return 1 to 9, spread across the three instances. Any instance can answer any request, so the apps are stateless. Redis is now the one process that holds state, and the one process that can’t fail.

Stopping an app

Next I stopped one app and measured each request:

docker compose stop app2
for i in $(seq 1 30); do
  echo -n "$i: "
  curl -s -o /dev/null -w "%{time_total}s\n" localhost:8080/count
  sleep 1
done

-o /dev/null drops the response body and -w "%{time_total}s\n" prints the total time. echo -n prints the request number without a newline, so the number and the time share a line.

Prediction: no request fails, because Nginx moves on to another app after app2 times out. I expected that timeout to be Nginx’s 60-second default.

3: 0.003316s
4: 3.059209s
5: 0.004764s
...
17: 0.004038s
18: 3.084284s
19: 0.004455s

No request failed, but the wait was 3 seconds, not 60.

There are three mechanisms here.

Why 3 seconds. A stopped container’s IP address disappears. To send a packet to an address on the local network, the kernel first needs its hardware address, so it sends an ARP request. Linux asks three times, one second apart. With no answer, it returns error 113, No route to host. The connect attempt fails with an error. Nginx’s 60-second proxy_connect_timeout applies only when nothing comes back at all, such as when a firewall drops packets silently.

Why nothing failed. proxy_next_upstream defaults to error timeout: on either, Nginx sends the same request to the next server. That is safe here because the request never reached app2, so no INCR ran twice.

Why only every 14th request. The defaults max_fails=1 and fail_timeout=10s mark app2 as unavailable for 10 seconds after one failure. Nginx learns this only from real requests, which makes it a passive health check. After 10 seconds, the next request on app2’s turn pays the 3 seconds again.

How fast a failure is detected depends on how the node fails:

The node is equally dead in all three rows. A failure shows up as delay before it shows up as an error, and from outside, a dead node and a slow node look the same.

Failing over between apps was easy because they hold no state: app1 and app3 can serve anything app2 could. Redis holds state, so traffic can only move to another Redis if that Redis has the same data. That is what replication is for.

3. A Redis follower

The change

Compose gets a second Redis that copies the first:

services:
  app1:
    build: ./app
    hostname: app1
    environment:
      PORT: 3000
      REDIS_LEADER_HOST: redis
      REDIS_FOLLOWER_HOST: redis-replica

  # app2 and app3 get the same line

  redis:
    image: redis:7

  redis-replica:
    image: redis:7
    command: redis-server --replicaof redis 6379
    depends_on:
      - redis

--replicaof redis 6379 tells the new process to connect to redis and copy everything from it.

Checking replication before writing code

I tested the infrastructure first, so a later failure would point at my code and not at the setup:

docker compose up --build -d
docker compose logs redis-replica
docker compose exec redis redis-cli SET hello world
docker compose exec redis-replica redis-cli GET hello
docker compose exec redis-replica redis-cli SET hello again

The replica’s log showed a full sync on its first connection: no earlier copy, so the leader sent a snapshot. The GET on the replica returned world, and the SET on the replica was refused:

(error) READONLY You can't write against a read only replica.

After the snapshot, the leader sends every write command down the same connection, and the follower runs it. Replication copies commands, not final values. A DEL on the leader reaches the follower the same way an LPUSH does.

The leader replies to the client before the follower has the write. That makes the replication asynchronous, and the gap between the two moments is the replication lag. On one laptop it is under a millisecond.

Posts, from the leader or the follower

The app gets a second client and two endpoints. The follower client is a copy of the leader client with a different host:

const app = express();
const PORT = process.env.PORT || 3000;

app.use(express.json());

const leader = new Redis({
  host: process.env.REDIS_LEADER_HOST ?? "redis",
  port: 6379,
  enableOfflineQueue: false,  // no connection? fail now, do not queue
  maxRetriesPerRequest: 1,     // retry a command one time only
  commandTimeout: 1000,        // no reply in 1 s? fail
});

const follower = new Redis({
  host: process.env.REDIS_FOLLOWER_HOST ?? "redis-replica",
  port: 6379,
  enableOfflineQueue: false,  // no connection? fail now, do not queue
  maxRetriesPerRequest: 1,     // retry a command one time only
  commandTimeout: 1000,        // no reply in 1 s? fail
});

leader.on("error", (err) => console.error("redis:", err.message));
follower.on("error", (err) => console.error("redis-replica:", err.message));

Each client has its own TCP connection, so each can fail on its own, and each needs its own error listener. express.json() is middleware that reads the request body stream, parses it as JSON and puts the result in req.body. Express doesn’t read bodies by default.

POST /posts writes to the leader. LPUSH adds to the left of the list and returns the new length, so the newest post comes first:

app.post("/posts", async (req, res) => {
  try {
    const text = req.body?.text;
    if (typeof text !== "string" || text === "") {
      return res.status(400).json({
        message: "Text is missing",
        instance: os.hostname()
      })
    }

    const len = await leader.lpush("posts", text);
    res.json({
      instance: os.hostname(),
      len,
      servedBy: "leader"
    });
  } catch (e) {
    const message = e instanceof Error ? e.message : String(e);
    res.status(503).json({
      message: "An error occurred: " + message,
      instance: os.hostname()
    });
  }
})

GET /posts reads from the follower unless ?from=leader asks for the leader. It decides once and uses that one value for both the client and the label:

app.get("/posts", async (req, res) => {
  try {
    const servedBy = req.query.from === "leader" ? "leader" : "follower";
    const client = servedBy === "leader" ? leader : follower;
    const posts = await client.lrange("posts", 0, -1);
    res.json({
      instance: os.hostname(),
      posts,
      servedBy
    })
  } catch (e) {
    const message = e instanceof Error ? e.message : String(e);
    res.status(503).json({
      message: "An error occurred: " + message,
      instance: os.hostname()
    });
  }
})

LRANGE posts 0 -1 returns the whole list: 0 is the first item and -1 the last. /count got the same catch block, so errors now return e.message and no longer {}.

Comparing log positions

Each node counts how many bytes of the write stream it has handled. That count is the replication offset, the position in the log. With three posts written, I read it on both nodes:

docker compose exec redis redis-cli INFO replication
docker compose exec redis-replica redis-cli INFO replication

Prediction: both offsets are equal.

role:master
connected_slaves:1
slave0:ip=172.19.0.6,port=6379,state=online,offset=566,lag=0
master_replid:910d04af7d565d360cd818c733ae9b23414db3bf
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:566
second_repl_offset:-1
repl_backlog_size:1048576

role:slave
master_link_status:up
master_last_io_seconds_ago:6
slave_repl_offset:580
master_replid:910d04af7d565d360cd818c733ae9b23414db3bf
master_repl_offset:580

The follower was 14 bytes ahead of the leader, which should be impossible.

When a leader has replicas and no writes are happening, it puts a PING into the replication stream every 10 seconds, the repl-ping-replica-period setting. In the Redis wire format, PING is *1\r\n$4\r\nPING\r\n: 4 + 4 + 6 = 14 bytes. It moves the offset forward on both nodes, like a write. A PING landed between my two commands.

I then predicted that if the readings differed again, the leader would be 14 bytes ahead. That was wrong too. My command read the leader first, so a PING between the two reads always makes the follower show more. The node read later has seen more time pass.

Three more facts from these fields:

• slave0:...offset=566 is the position the follower confirmed to the leader. The follower sends that confirmation every second, and it matched the leader’s 566 at that moment.
• master_last_io_seconds_ago:6 means the follower heard the last PING six seconds earlier. If it hears nothing for 60 seconds, the repl-timeout, it drops the link.
• Reads never move an offset. Only writes enter the stream.

There is no clock that stops both machines while I look. Each answer is that node’s state at the moment it answered.

Stopping replication

To see lag, I made it permanent:

docker compose exec redis-replica redis-cli REPLICAOF NO ONE

This closes the replication link and turns the follower into an independent master that accepts writes.

Predictions: role:master on the replica. connected_slaves:0 on the leader. After 30 seconds with no writes, the offsets move apart. The replica has no master_replid of its own, since it is the master now.

The first two were right. The offsets didn’t move at all: with no replicas, the leader sends no PING, because PING exists only to keep a replication link alive. The ID prediction was wrong:

role:master
connected_slaves:0
master_replid:fa2322728489986297fd0837a74ab6e1fbc7d72d
master_replid2:910d04af7d565d360cd818c733ae9b23414db3bf
master_repl_offset:1000
second_repl_offset:1001

The node created a new ID, fa23…, and moved the old one, 910d…, into master_replid2. second_repl_offset:1001 is the split point: bytes 1 to 1000 are shared with 910d…, and anything from byte 1001 on belongs only to fa23….

A replication ID names a history, one continuous sequence of writes, not a node. Promotion starts a new history, and keeping the old ID lets the node prove what it shares with the old one, up to the split.

Then I wrote two posts and read from both nodes:

for t in four five; do curl -s -X POST localhost:8080/posts -H 'Content-Type: application/json' -d "{\"text\":\"$t\"}"; echo; done
curl -s "localhost:8080/posts?from=follower"; echo
curl -s "localhost:8080/posts?from=leader"; echo

{"instance":"app1","len":1,"servedBy":"leader"}
{"instance":"app2","len":2,"servedBy":"leader"}
{"instance":"app3","posts":[],"servedBy":"follower"}
{"instance":"app1","posts":["five","four"],"servedBy":"leader"}

My list happened to be empty at the moment of the split, so the follower had nothing at all.

For a user who posts “five” and refreshes, the post vanishes. The follower violated read-your-writes: a user should always see their own writes.

Read-your-writes with a cookie

The fix: for a short time after a user writes, send that user’s reads to the leader. Everyone else still reads from the follower.

The app has to remember “this user wrote at time T”. A variable in the app is step 1’s mistake again: the write can land on app1 and the next read on app3. Storing it in Redis works, but adds a Redis call to every read. A cookie keeps the state in the client, and the client sends it with every request, so any instance can read it.

A cookie is two HTTP headers. The response carries Set-Cookie: lastWrite=1791572060182, and the client sends Cookie: lastWrite=1791572060182 on later requests. Browsers do this on their own. curl needs a cookie jar: -c jar.txt saves cookies and -b jar.txt sends them.

The window length goes in the compose file, so it can be tuned without touching the code:

  app1:
    build: ./app
    hostname: app1
    environment:
      PORT: 3000
      REDIS_LEADER_HOST: redis
      REDIS_FOLLOWER_HOST: redis-replica
      RYW_WINDOW_MS: 20000

  # app2 and app3 get the same line

bun add cookie-parser adds middleware that parses the Cookie header into req.cookies. The code changes:

import Redis from "ioredis";
import cookieParser from "cookie-parser";

const app = express();
const PORT = process.env.PORT || 3000;
const RYW_WINDOW_MS = Number(process.env.RYW_WINDOW_MS ?? 10000);

app.use(express.json());
app.use(cookieParser())

POST /posts sets the cookie after a successful write. A failed write has nothing to read back:

    const len = await leader.lpush("posts", text);
    res.cookie("lastWrite", String(Date.now()), { httpOnly: true, maxAge: RYW_WINDOW_MS })

maxAge makes the client drop the cookie when the window ends, and httpOnly keeps browser JavaScript from reading or changing it.

GET /posts now chooses the node in a fixed order:

    const servedBy = req.query.from === "leader" ? "leader" : "follower";
    // Choose one time, so the client and the label always agree
    let servedBy: "leader" | "follower";
    if (req.query.from === "leader") {
      servedBy = "leader";
    } else if (req.query.from === "follower") {
      servedBy = "follower";
    } else {
      // A bad or missing cookie gives NaN, and NaN < anything is false
      const lastWrite = Number(req.cookies.lastWrite);
      const recentWrite = Number.isFinite(lastWrite) && Date.now() - lastWrite < RYW_WINDOW_MS;
      servedBy = recentWrite ? "leader" : "follower";
    }
    const client = servedBy === "leader" ? leader : follower;

?from= comes first, so ?from=follower reads the follower even right after a post. I needed that to inspect the follower directly.

The cookie value is a string, so Number() turns it into a timestamp. A missing or malformed cookie gives NaN, and every comparison with NaN is false, so Number.isFinite treats it as “no recent write”. RYW_WINDOW_MS also goes through Number() once at the top, because environment variables are strings.

Replication was still stopped. I posted hello with the cookie jar, read right away, then waited past the 20-second window and read again.

curl -c jar.txt -b jar.txt -X POST localhost:8080/posts -H 'Content-Type: application/json' -d '{"text":"hello"}'; echo
curl -c jar.txt -b jar.txt localhost:8080/posts; echo

Inside the window, reads came from the leader and included hello. After it, reads went back to the follower, and hello was gone.

The cookie assumes the follower catches up within the window. Normally the lag is milliseconds, so 20 seconds is generous. Here the lag never ends, so the assumption fails. The fix has three limits:

• It works only while the lag is shorter than the window. The window hides the lag. It doesn’t end it.
• Clocks differ between machines. Date.now() runs on the app that handled the POST and is compared on the app that handles the GET. On my laptop all three share one clock. If app1’s clock ran 30 seconds ahead, app1 would see the write as older than it is, close the window early and serve stale data.
• It protects only the writer. Another user reading a new post a second late is normal eventual consistency. A worse case is monotonic reads: that user sees the post, refreshes, and it disappears, because the second read hit a follower further behind than the first. That needs at least two followers with different lag, so this setup can’t show it.

Reconnecting the follower

docker compose exec redis-replica redis-cli REPLICAOF redis 6379
docker compose logs redis-replica | tail -20
docker compose logs redis | tail -20

When a follower connects, it sends PSYNC <replid> <offset+1>: “continue history X from byte N”. If the leader can prove it holds the same history up to that byte, and those bytes are still in its 1 MB backlog, it sends only the missing part. That is a partial resync. Otherwise it sends a snapshot, and the follower deletes its own data first. That is a full resync.

Prediction: the follower sends 910d…, which the leader knows, so this is a partial resync.

The log said full resync. I had rebuilt the stack before this run, so the IDs differed from the ones above, but the logic was the same:

• After REPLICAOF NO ONE, the follower’s current ID was its new one, fa23… in the earlier run. A node sends its current master_replid, not replid2.
• The leader had never been promoted, so its IDs were its own and 0000…. Nothing matched.
• The leader had no proof that the follower’s bytes were the same as its own, so it sent a snapshot.

The follower never accepted a write while it was a master, but Redis doesn’t check that. It could have, so its data proves nothing.

The ID trick works in one direction only. A new leader keeps the old ID in replid2, so followers of the old history can continue from it. The old leader never learns the new ID. Step 4 uses the useful direction.

4. Manual failover

A failover has three jobs:

1. Detect that the leader is dead.
2. Choose a new leader.
3. Reconfigure every client to use it.

Sentinel does all three automatically. Here I did them by hand. Step 4 starts as a copy of step 3, with replication running and two posts written.

Detect

docker compose stop redis
curl -s -X POST localhost:8080/posts -H 'Content-Type: application/json' -d '{"text":"after-crash"}'; echo
curl -s 'localhost:8080/posts?from=follower'; echo
docker compose exec redis-replica redis-cli INFO replication

Predictions: the POST returns 503. Reads from the follower still work, without after-crash. The follower can’t take writes because it is configured as a replica, and it will learn that the leader is down when the PINGs stop.

{"message":"An error occurred: Stream isn't writeable and enableOfflineQueue options is false","instance":"app2"}
{"instance":"app3","posts":["post2","post1"],"servedBy":"follower"}

role:slave
master_link_status:down
master_last_io_seconds_ago:-1
master_link_down_since_seconds:341
slave_repl_offset:223
master_replid:cbcc1c979e509c20194dd72c9ec1b3dad16443e3
master_replid2:0000000000000000000000000000000000000000

The first two predictions held. The write failed in milliseconds because ioredis had no connection and enableOfflineQueue: false rejected the command instead of queueing it. Reads still worked, so the system was read-only.

The third was half right. The follower didn’t need the PINGs: when the Redis process stopped, the operating system closed the TCP connection, and the link went down at once. But 341 seconds later, far past the 60-second repl-timeout, it was still role:slave. Redis never promotes a replica by itself.

The reason is that one node can’t tell these two cases apart:

A follower that promoted itself on silence would be right in the first case. In the second, there would be two leaders, each taking different writes. Silence proves only that this node can’t reach the leader. Real systems either ask several observers and act only when a majority agrees, as Sentinel does, or leave the decision to a person.

Choose

docker compose exec redis-replica redis-cli REPLICAOF NO ONE

The step 3 pattern again: a new ID d846…, the old ID cbcc… in replid2, and second_repl_offset:224, one past the replica’s last byte, 223.

The replica now accepts writes, but every POST still failed. The apps still sent writes to redis. The new leader existed, and no client knew about it.

Reconfigure

The only change in step 4 is in the compose file:

  app1:
    build: ./app
    hostname: app1
    environment:
      PORT: 3000
      REDIS_LEADER_HOST: redis
      REDIS_LEADER_HOST: redis-replica
      REDIS_FOLLOWER_HOST: redis-replica
      RYW_WINDOW_MS: 20000

  # app2 and app3 get the same change

docker compose up -d app1 app2 app3

Leader reads and follower reads now go to the same node, so read-your-writes holds trivially. There is also only one copy of the data until a new follower joins. A failover removes redundancy.

The app3 surprise

After the reconfigure, reads aimed at app3 waited and were then answered by app1 or app2. app3 never appeared in the output. Nginx’s log:

connect() failed (113: No route to host) while connecting to upstream, request: "GET /posts?from=follower HTTP/1.1", upstream: "http://172.19.0.5:3000/posts?from=follower"

The current addresses:

app1 172.19.0.3
app2 172.19.0.4
app3 172.19.0.2
redis-replica 172.19.0.6
nginx 172.19.0.7

I had skipped docker compose restart nginx. up -d recreated the app containers, and Docker handed out addresses again:

Nginx still had .5 from its start-up lookup. Each request on app3’s turn hit the step 2 path: no ARP reply, 3 seconds, error 113, retry on another app.

docker compose restart nginx made Nginx resolve the names again:

{"instance":"app1","posts":["after-failover","post2","post1"],"servedBy":"follower"}
{"instance":"app2","posts":["after-failover","post2","post1"],"servedBy":"follower"}
{"instance":"app3","posts":["after-failover","post2","post1"],"servedBy":"follower"}

I had changed the Redis address correctly, but a second component also remembered addresses. Every one of them has to update during a failover: the apps, the proxies and any DNS cache. This is service discovery in miniature.

A trap in the compose file

The compose file still says command: redis-server --replicaof redis 6379 for redis-replica. REPLICAOF NO ONE changed only the running process.

Prediction: if the containers are recreated, there is one leader and one follower again, which is fine.

The direction makes it not fine. A recreated redis-replica would follow redis, the old leader, which doesn’t have the posts written after the failover. It would full-resync from it and delete those posts. The failover would reverse itself and lose data with no error. up -d only recreates a service whose configuration changed, so this happens after down and up, or --force-recreate, or an edit to that service. Runtime state and declared configuration have to agree, or the next restart restores the old setup.

Split brain

The old leader comes back:

docker compose start redis
docker compose exec redis redis-cli LRANGE posts 0 -1

Now two nodes report role:master. The apps are unaffected, since all three point at redis-replica. In a real system, though, something always keeps the old address: an app nobody updated, a cron job, an admin script. I simulated one by writing directly to the old leader:

docker compose exec redis redis-cli LPUSH posts stale-client-write

Then I made the old leader a follower of the new one:

docker compose exec redis redis-cli REPLICAOF redis-replica 6379
docker compose logs --tail=20 redis-replica
docker compose exec redis redis-cli LRANGE posts 0 -1

Predictions, and what happened:

1. After start, redis has post1 and post2 but not after-failover. Right. stop sends SIGTERM, Redis saves a snapshot to disk before exiting, and start loads it. It slept through the failover.
2. Two masters is a problem by itself. Partly. Two masters is a risk, but nothing is lost while nobody writes to the old one. The damage starts at the first write that reaches the wrong master. After that, the two histories differ, and one has to lose.
3. The resync is a full one, because the old leader gets a new ID when it becomes a replica. The result was right and the reason wrong. REPLICAOF host port doesn’t create an ID. Only becoming a master does.
4. stale-client-write is gone, and its client never got an error. Right. The stale client got (integer) 3, a success, and Redis really had stored the write. The resync deleted it.

The new leader’s log gave the real reason for the full resync:

Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'c701e8291f5eeed645d19382a962568e0b57f001', my replication IDs are 'd846592190edcd3781fb4f7b17c9aa26d8a461d9' and 'cbcc1c979e509c20194dd72c9ec1b3dad16443e3')

The restarted redis came back with a new ID, c701…. A node that starts as a master can’t prove that its snapshot file matches what its replicas saw. After a crash, the file may be minutes old. Someone may have replaced it with a backup. Keeping cbcc… would promise a shared history it can’t check, so it starts a new one. Redis does store the ID in the snapshot, but only reuses it when the node starts as a replica and lets the leader decide.

The stale write would have forced a full resync anyway: it moved the old leader past byte 224, where its history and the new leader’s differ. Two separate reasons led to the same result.

This is the worst kind of data loss: an acknowledged write that was lost. The write wasn’t lost because the resync was a mistake. It was lost when it was written, because it went into a history that couldn’t win. The resync only made that visible. Asynchronous replication can lose acknowledged writes in the same way without any stale client: if the leader dies before a write reaches the follower, the promoted follower never had it.

The PSYNC rule

Every resync in this lab followed one rule. The follower sends its current ID and the next byte it needs. The leader accepts a partial resync only if:

• the ID is the leader’s replid, or the ID is its replid2 and the offset is at or before second_repl_offset,
• and that byte is still in the leader’s backlog.

Put another way, the follower’s history must be a prefix of the leader’s. The leader can add the bytes the follower is missing. It can’t remove bytes the follower has and it doesn’t, so extra bytes always mean a full resync, and the follower’s extra writes are deleted.

Redis keeps only two IDs: the current one and the one before the last promotion. Each promotion pushes the current ID into replid2, and the older replid2 is forgotten. A node that is two promotions behind gets a full resync even if its data is an exact prefix, because nobody can check it anymore. In practice the 1 MB backlog usually runs out first.

What I took away

1. Stateless services with shared state are the base for scaling. Any instance can serve any request only when no instance holds anything of its own.
2. Asynchronous replication means lag, and lag means stale reads. Read-your-writes doesn’t come for free. It needs a deliberate rule, like sending recent writers to the leader, and that rule has limits.
3. A replication ID and an offset name a position in a history. Catching up cheaply depends on proving a shared prefix, never on comparing data.
4. A failover has three jobs: detect, choose, reconfigure. Detection is the hard one, because a dead node and an unreachable node look the same.
5. Split brain is dangerous because writes succeed. They are deleted later, with no error to anyone.
6. Asynchronous replication can lose acknowledged writes during a failover. A write that only the old leader had is gone once the old leader rejoins.