Loading and Preparing Data

Ensign Demir

The raw sensor data comes in faster than we can process it in one pass. But this is just infinite scroll, right? Fetch everything once, hold the list, render a window. I pulled the full archive into one array overnight.

Cut to: Briefing

RangeError: Array buffer allocation failed. Engineering rig, 19,204 readings into a 28,800-reading archive. The crash takes ARIA's overnight diagnostic queue down with it. Queue rebuild: four hours. Logged against: Ensign Demir.

ARIA

Confirmed. Batch processing will allow me to analyze data incrementally while maintaining memory efficiency. I note that I said the same thing about my parsing rules.

Cut to: Briefing

One method call decides whether this pipeline survives contact with real sensor data: .batch(32). The model sees 32 readings at a time and pulls the next chunk only when it finishes the current one. Demir's instinct wasn't wrong about scrolling; it was wrong about what a well-built infinite scroll actually does. A good list never holds every page. It holds a window, and that's the discipline this lesson installs: load chunks, never everything.

From Fetch to Feed

In frontend development, you fetch data from APIs and render it. In ML, you fetch data and feed it to a model. The pattern is remarkably similar, and the key promise of a generator is the same as paginated fetching: nothing is loaded until it's asked for.

The key difference from frontend pagination: ML shuffles the data before batching. The shuffle(100) buffer holds 100 items and yields them in random order. This prevents the model from learning the order the data arrived in instead of the patterns inside it.

Why Batching Matters

Batching is virtual-list rendering for model input. You don't render 10,000 DOM nodes at once; you render a window of visible items. ML models likewise process a window (batch) of data at a time, updating their understanding incrementally.

Challenge

Build the lazy pipeline yourself: generator, shuffle, batch, without ever holding the archive in memory. The grader counts your reads.