Distributed systems with three boards is an Expert course in the Coding track. Every lesson runs the same C++ sketch on three boards, A, B and C, each with a lamp. Each works out who it is from its MAC address, and node A narrates on its serial monitor. It takes about six hours.
What you build, lesson by lesson
- Clocks and order: skewed wall clocks put a reply before its question. Lamport clocks put causes first, and you break their ties by node name to get one total order on every node.
- Messages and loss: numbered heartbeats, 25% loss on purpose, a failure detector, and the hard truth that a dead node and an unlucky one look the same. You make node A name every gap.
- Retries and idempotence: two clients add to a counter with acks and exponential backoff. Lost acks make retries count twice (25 instead of 20) until you make the server remember what it has applied.
- Leader election: the bully algorithm, with the leader's lamp lit, a crash, a comeback, and a stale leader after a partition that a term number teaches node A to ignore.
- Consensus light: ballots and votes decide what every lamp shows. Asking for every vote stalls when one board dies; asking for a majority keeps going, and the lesson shows why two majorities always overlap.
- Partitions: a visitor count gossiped between the boards. Split the network, count on both sides, heal it, and watch a naive merge forget three people until a slot-by-slot merge (a G-Counter CRDT) keeps all thirteen. Then CAP: consistent or available, and how to choose.
The boss: a fault-tolerant counter
The last lesson finishes the counter: slot-by-slot merges, whole-state gossip every 300 ms, a failure detector, and an agreement lamp on each board. You throw 40% loss, a crash and a partition at it and watch it agree every time, then add a quorum read that refuses to answer while a majority cannot confirm the value. The finale is a write-up of the model, the failures, the guarantees and the limits, with real output behind every claim.
What the simulation does, and what it does not
ESP-NOW frames really cross between the boards on the canvas
(ESP-NOW between boards), with no router and no wires
between them. The simulated air has no range, fading or noise, so on its own it
never loses a frame. Every failure in the course is a rule in the sketch: a
shared block called the desk network drops frames by a seeded dice roll, across
a partition, or to and from a crashed node. You switch those rules by typing
chaos commands on node A (loss 30, crash C, revive C, split A, heal),
and because the dice are seeded, every number in a lesson is the same on every
run.
A sketch edit rebuilds node A's sketch only, so every change you make is to something node A does, and B and C keep running the old code. The lessons say when that matters, and it is how a fleet looks halfway through a rollout.
Before you start
You should be fluent in C++ on a microcontroller and know what ESP-NOW is; Connected devices has a lesson on it. Design a device API pairs well, for the client side of retries and idempotence.