Lessons and the learning game

Device to cloud: telemetry that survives

Get every reading to the server once and in order through outages and reboots: sequence numbers, batching, backoff, an SD queue and idempotent uploads.

The bench

Every lesson opens one telemetry node: an ESP32-C3 with a BME280 sensor on I2C (SDA GPIO 8, SCL GPIO 9) and a microSD card on SPI (SCK 4, MISO 5, MOSI 6, CS 7). The cloud is Mokxi's simulated network: uploads go to api.mokxi, whose status and body are set in the Cloud panel (the lessons' checks set them for you to make outages on purpose), and echo.mokxi returns a payload byte for byte. See WiFi in Mokxi.

Take the course at /learn/path/iot-data-pipelines on /learn.

The lessons

  1. Telemetry design. A device id, a sequence number never reused, an uptime timestamp and units in every name; what a day of readings costs.
  2. JSON payloads. ArduinoJson documents, nested objects and arrays, exact decimals, measuring before sending, short keys, and a reply parsed with every failure checked.
  3. Batching. Many readings per request, each keeping its seq, and a batch kept whole when the post fails.
  4. Backoff. Exponential backoff to a cap with full jitter, a reset on success, and no retries for a 4xx.
  5. An offline queue. Readings appended to a JSON Lines file on the card before sending, uploaded oldest first, marked sent only after a 2xx, and a sequence number that survives a reboot.
  6. Device identity. An id from a config file or the MAC, an API key kept out of the code, an Idempotency-Key per upload, and a stop on a refused key.
  7. Boss: a reliable telemetry node. All of it together, through an outage, a reboot and a refused key.

What each sketch does

Sketch What it does
dptelem Sends readings with id, seq, uptime and units, checked against the echo.
dpjson Builds, measures and posts an ArduinoJson payload, parses the reply.
dpbatch Batches readings into one JSON array per request.
dpbackoff Retries with exponential backoff, a cap and full jitter.
dpqueue An SD card queue that survives outages and reboots.
dpident Identity from the card or the MAC, an API key, an Idempotency-Key.
dpnode The reliable telemetry node.

Honest limits

  • The network is simulated: no radio and no real internet. api.mokxi answers what the Cloud panel says; it does not store readings or deduplicate by Idempotency-Key, so the lessons explain the server's half rather than show it.
  • The HTTP client here does not return response headers, so Retry-After is explained, not read.
  • The card uses 8.3 file names (QUEUE.TXT, SEQ.TXT, SENT.TXT, CONFIG.TXT), and SD.h's File has no readStringUntil(), so the sketches read lines by hand.

Wireless links: nRF24 and ESP-NOW.