Skip to content
Idaho AI Classroom
All lessons
AI Technical AptitudeData and Analysis9-12.AITA.DA.3

Collect, Split, Then Train

9–12 CS / engineering50 minutes90 minutes (lab extension)

Standard quoted exactly

Examine the procedure of how data is used to train machine learning models.

Clarification. The lifecycle of collecting and munging data, identifying features, and creating train/dev/test data splits.

Student-friendly learning targets

  • I can walk a public CSV from collection notes through munging, feature choices, and a train/dev/test split.
  • I can tell a feature from a target and explain why the test rows stay sealed.
  • I can split a small Idaho table by hand without leaking the test set into training.

Essential questions

  1. If the model has already seen the test rows, what is left to test?
  2. Which columns are features we might use, and which column is the thing we want to predict?
  3. What gets lost when we skip munging and jump to 'train'?

Objectives

  1. Station-lab a potato-yield or wildfire CSV: source, mung, features, splits.
  2. Identify target vs. candidate features and drop at least one leaky or identifier column.
  3. Create a train / dev / test split (e.g., 70/15/15) with a documented rule (random with seed, or by year, or by county).
  4. Explain in one sentence why collection and munging happen before any training.

Key vocabulary

Munging
The unglamorous cleaning and reshaping of raw tables so rows and columns can be used: types, missing values, units, duplicates.
Feature
An input column the model is allowed to use (April rainfall, county, seed variety). Not the answer column.
Target
The column we want to predict (yield in cwt/acre, or did this grid cell ignite).
Train / dev / test split
Three disjoint slices: train fits the model, dev (validation) tunes choices, test is used once to report honesty.
Leakage
Letting the model see information it would not have at prediction time, including test rows or future yields.

Teacher background

This is the data-lifecycle lab, not the methods lecture. AIM.8 covers the same procedure from a methods angle (pipeline diagram, feature engineering, leakage card-sort). DA.3 is stations with a real-feeling public CSV in students' hands. Use Idaho potato yield by county-year (USDA NASS public) or a simplified wildfire-start table (grid_id, date, temp, RH, wind, ignition). Students do not train a model. They collect provenance, mung a few dirty rows, name features vs. target, then physically split colored row cards into train/dev/test. Emphasize sealing the test envelope. Spreadsheet FILTER/RAND is enough; Python train_test_split is an extension. Do not spend this period on backprop, trees, or XOR — that is DA.6 / AIM.5.

Materials and prep

Materials

  • Station kits (4): COLLECT, MUNG, FEATURES, SPLIT. Printed CSV of 40 rows (potato yield by Idaho county-year or wildfire weather + ignition).
  • Row cards (one card per row) in an envelope for the split station; three bins labeled TRAIN / DEV / TEST.
  • Spreadsheet copy of the same CSV; printed version for offline.
  • Provenance slip: dataset name, public URL or agency (USDA NASS / NIFC-style), date retrieved, license/public-domain note.
  • Feature sticky pads: keep / drop / not sure.
  • Teacher key with a recommended target (yield or ignition) and two leaky columns (final_harvest_notes, post_fire_acres — not known at prediction time).

Before class

  • Build the 40-row public-style table with 4 dirty rows (n/a yield, mixed county names, duplicate year, a future-season leak column).
  • Set up four physical stations; 90-second rotation timer. For a short period, run two rounds of 8 minutes.
  • Print AIM.8 cross-reference on the teacher desk: if students ask about feature engineering depth or leakage types, park it ('methods view is AIM.8').
  • Cut row cards and shuffle. Put a 'TEST — do not open until the end of the unit' sticker on the test bin.

Instructional sequence

Would you grade your own homework?

5 min
  1. Ask: a student studies the answer key, then takes the same quiz. Fair test of learning?
  2. Map it: training on all rows then 'testing' on those rows is the answer-key trick.
  3. Preview stations: collect, mung, features, split — then, and only then, would someone train.

Lifecycle in one pass

8 min
  1. Board the sequence: collect (where from, who measured) → mung (DA.1–DA.2 skills) → identify features and target → split train/dev/test → (later) train. Circle that we stop before train.
  2. Define target vs. feature with yield: target is cwt_per_acre; county, rain_apr, seed_variety can be features; farm_id is an identifier; post_harvest_price is a leak if unknown at planting.
  3. Draw 70/15/15 boxes. Test is a sealed envelope. Dev is for trying cleaning or model choices. Train is the only place a future algorithm may look repeatedly.

Name the target on row one

8 min
  1. Project the CSV header. Class votes: which column is the target for a yield model?
  2. Mark two columns as leaks or IDs and say why they must not be features.
  3. Demonstrate one mung: collapse Canyon / canyon / Canyon Co. and recode n/a yield to missing, not zero.

Station pass — collect and mung

8 min
  1. Half the pairs start at COLLECT (fill provenance slip) then MUNG (fix two dirty rows on paper or sheet).
  2. The other half start at FEATURES (keep/drop stickies) then SPLIT (deal row cards into 70/15/15, write the rule).
  3. Rotate once so every pair hits all four before hands-on wrap. Offline: the printed CSV is the station.

Real-world examples

  • USDA NASS Idaho potato production by county is collected by survey, not by a model; if you skip provenance you will double-count a revised year.
  • A wildfire-start table that includes final containment acres as a feature is leaking the future — crews do not know that on the ignition morning.
  • Energy-load models at Idaho Power split by time: you train on past days and test on later days, not a random shuffle that puts Tuesday afternoon in both.
  • University of Idaho variety trials: seed lot IDs look like features but only name the row; using them lets a model memorize plots instead of weather.

Hands-on activity

Finish the four stations

16 min
  1. COLLECT: complete agency, year range, public-use note, and one thing the CSV does not measure (e.g., irrigation method).
  2. MUNG: log each dirty row and the DA.2-style response (flag, recode, quarantine).
  3. FEATURES: list target, at least four candidate features, and one dropped leak/ID.
  4. SPLIT: place cards, record counts, write 'test envelope stays closed.' Photograph or tally bins. No model is trained.

Discussion questions

  1. Random 70/15/15 vs. split by year: which is honest for predicting next season's yield?
  2. If Canyon County has twice as many rows as Clark County, what happens to a random split?
  3. Why do we need three piles, not two (train and test)?
  4. What would count as leakage in a river-gauge forecasting table?

Differentiation

Support

  • Target column pre-highlighted; students only choose features and place cards into bins with printed 70/15/15 counts (28/6/6 of 40).
  • Mung station limited to two error types already seen in DA.2.

Challenge

  • Design a stratified split so each of four counties appears in train, dev, and test. Show the counts.
  • Find a leakage column the teacher did not flag and argue it.

Multilingual learners

  • Station header cards with verbs: collect / limpiar (mung) / features (entradas) / split (partir).
  • Allow provenance slip in the student's strongest language; column names stay English as in the CSV.

IEP / 504

  • Stay at two stations if four is overload: FEATURES and SPLIT are the standard's core.
  • Large row cards and bins; a partner may handle the tally while the student decides keep/drop.

Assessment

Formative

  • Provenance slip and keep/drop stickies at stations.
  • Teacher checks that test cards are not mixed back into train.

Summative

  • Station packet: source note, mung log, feature list with target, split counts and rule, one leakage sentence.
  • No points for training a model; this lab ends at the split.

Success criteria

  • Student names a target and at least three legal features from the CSV.
  • Student produces disjoint train/dev/test counts that add to all rows.
  • Student can explain why the test bin stays sealed.

Responsible use, ethics, and privacy

Responsible use

Public NASS/NIFC-style tables only. Do not collect classmate farm, address, or family yield data. The 'collect' station is provenance of an existing public file, not a new student survey.

Ethics

A split that leaves every small county in train and every large county in test (or the reverse) will lie about performance. Who is in which bin is a fairness issue, not just a math one.

Privacy

FERPA: no student names on row cards. If you ever localize with a co-op dataset, strip names, phone numbers, and section-township-range that identifies a family farm before it enters the room.

Reflection

  1. Where in the four stations did you almost skip a step, and what would have broken?
  2. How will you explain 'sealed test' to someone who just wants to train?
  3. What did this CSV not collect that a grower would care about?

Homework

On the printed 12-row mini-table, circle the target, list three features, cross out one leak, and assign each row T, D, or S using a 8/2/2 split. Write the split rule in one line. No training.

Closing

Hold the TEST envelope: this is the honest exam. We collected, munged, named features, and split. We did not train — on purpose. AIM.8 will show the same pipeline from the methods side, including feature engineering and leakage traps.

Extensions and cross-curricular links

Go further

  • 90-minute block: second rotation with the wildfire CSV if potato was first (or the reverse); compare whether a random split is honest for time-based ignition.
  • Python extension: sklearn train_test_split twice to make train/dev/test; print value_counts by county to show a bad shuffle.
  • Do not duplicate AIM.8: if you have extra time, deepen the CSV work, not a lecture on feature engineering.
  • Printed tracing: number each row 1–40 and color train/dev/test; hang the sealed test list.
Statistics
Train/dev/test is a cousin of train/test and holdout; sampling design (by county vs. random) changes estimates.
Agriculture
Yield tables come from surveys and trials; collection quality caps any later model.
Geography
County and year are spatial-temporal units; splitting without geography can test on the wrong place.