Lessons and the learning game

Distributed systems with three boards

Three ESP32-C3 boards talking ESP-NOW teach clocks, loss, retries, elections, consensus and partitions, ending in a counter that survives them all.

Distributed systems with three boards is an Expert course in the Coding track. Every lesson runs the same C++ sketch on three boards, A, B and C, each with a lamp. Each works out who it is from its MAC address, and node A narrates on its serial monitor. It takes about six hours.

What you build, lesson by lesson

  • Clocks and order: skewed wall clocks put a reply before its question. Lamport clocks put causes first, and you break their ties by node name to get one total order on every node.
  • Messages and loss: numbered heartbeats, 25% loss on purpose, a failure detector, and the hard truth that a dead node and an unlucky one look the same. You make node A name every gap.
  • Retries and idempotence: two clients add to a counter with acks and exponential backoff. Lost acks make retries count twice (25 instead of 20) until you make the server remember what it has applied.
  • Leader election: the bully algorithm, with the leader's lamp lit, a crash, a comeback, and a stale leader after a partition that a term number teaches node A to ignore.
  • Consensus light: ballots and votes decide what every lamp shows. Asking for every vote stalls when one board dies; asking for a majority keeps going, and the lesson shows why two majorities always overlap.
  • Partitions: a visitor count gossiped between the boards. Split the network, count on both sides, heal it, and watch a naive merge forget three people until a slot-by-slot merge (a G-Counter CRDT) keeps all thirteen. Then CAP: consistent or available, and how to choose.

The boss: a fault-tolerant counter

The last lesson finishes the counter: slot-by-slot merges, whole-state gossip every 300 ms, a failure detector, and an agreement lamp on each board. You throw 40% loss, a crash and a partition at it and watch it agree every time, then add a quorum read that refuses to answer while a majority cannot confirm the value. The finale is a write-up of the model, the failures, the guarantees and the limits, with real output behind every claim.

What the simulation does, and what it does not

ESP-NOW frames really cross between the boards on the canvas (ESP-NOW between boards), with no router and no wires between them. The simulated air has no range, fading or noise, so on its own it never loses a frame. Every failure in the course is a rule in the sketch: a shared block called the desk network drops frames by a seeded dice roll, across a partition, or to and from a crashed node. You switch those rules by typing chaos commands on node A (loss 30, crash C, revive C, split A, heal), and because the dice are seeded, every number in a lesson is the same on every run.

A sketch edit rebuilds node A's sketch only, so every change you make is to something node A does, and B and C keep running the old code. The lessons say when that matters, and it is how a fleet looks halfway through a rollout.

Before you start

You should be fluent in C++ on a microcontroller and know what ESP-NOW is; Connected devices has a lesson on it. Design a device API pairs well, for the client side of retries and idempotence.