Lessons and the learning game

Data analysis with plain Python

Do by hand what pandas does, on an ocean buoy's log: load, clean, describe, group, chart and correlate with nothing but Python's standard library.

The course

Data analysis with plain Python is an intermediate Coding course in seven lessons on a Raspberry Pi 4. The data is Buoy K7, a weather buoy off the coast: a month of daily sea temperature, wave height and wind in a buoy.csv tab beside main.py, then a whole year of weekly readings in year.csv. Every step is checked against what the Pi prints, and every number in the lessons is one the program really prints.

  1. Loading data: lines, then rows with csv.reader, numbers with int() and float(), the ValueError a blank cell causes and how to survive it, rows by column name with csv.DictReader, and min and max catching the buoy's 999.0 sentinel.
  2. Cleaning: stray spaces and lowercase, blanks turned into None, range checks that catch sentinels, a repeated day thrown out, a clean file written with csv.writer, and the cleaner moved into its own buoy.py module.
  3. statistics: mean against median and why one storm splits them, stdev against pstdev (n - 1 and samples), z-scores that flag day 17, and a summary table lined up with f-string widths.
  4. Grouping with Counter: collections.Counter and most_common, shares printed as percentages, waves binned into the Douglas sea scale, a mean wave per wind direction, and windy days per week.
  5. Text charts: bar charts from "#" * n, bars scaled to a width, a histogram of wave heights, a time series turned on its side, and sparklines that fit a month on one line.
  6. Correlation: a scatter plot in text, covariance, Pearson's r written from its definition, the upwelling that makes a windy day cool the sea the day after, and what one bad reading does to r.
  7. Project: a year of ocean weather: the finale.

The finale

The project runs the whole course on 52 weeks of Buoy K7. It loads and cleans the year (three gaps, one of them a sentinel), compares the four seasons (spring is the windiest and has the coldest sea, at 10.9 C), charts the sea temperature in four-week blocks, correlates wind, waves, sea and rain, and finds the records, down to a 20 week summer dry spell.

Then the data goes live. The button on GPIO2 is a tipping-bucket rain gauge, the kind real buoys carry: a little seesaw that tips once per 0.2 mm of rain and closes a switch. A gpiozero Button with bounce_time counts the tips, and the LED on GPIO17 is a storm lamp that lights when today's rain passes an average wet-week day, a threshold worked out from the year's own data (3.31 mm, so the lamp lights on tip 17).

What the Pi's engine does differently

The Pi runs MicroPython 1.29, not CPython, with a CPython-style standard library. For this course that shows in a few places:

  • There is no numpy or pandas, which is the point: csv, statistics and collections are all here.
  • statistics has mean, fmean, median, mode, variance, stdev and pstdev, but not correlation (CPython added it in 3.10), so the course writes Pearson's r itself.
  • A dict does not keep the order keys went in, so every table prints in an order the program chooses (a tuple of names, sorted, or a range).
  • str.ljust and str.rjust are missing; f-string widths like {name:<6} and {value:>7.2f} line up columns on both engines.
  • int("") stops with ValueError: invalid syntax for integer with base 10, worded differently from CPython but the same exception.
  • A file the program writes, like clean.csv, lasts while the program runs, since the Pi on the canvas has no SD card.