Skip to content
Idaho AI Classroom
All lessons
AI Technical AptitudeAI Methods9-12.AITA.AIM.8

The Lifecycle, Methods View

9–12 CS / engineering50 minutes90 minutes (lab extension)

Standard quoted exactly

Examine the procedure of how data is used to train machine learning models.

Example from the standards. Looking at the complete data lifecycle including collecting, data munging, feature engineering, and creating training, development/validation, and test data splits.

Student-friendly learning targets

  • I can diagram the full training procedure: collect, mung, engineer features, split train/dev/test, train, evaluate.
  • I can explain feature engineering and data leakage in methods language.
  • I can contrast this lecture+card-sort with DA.3's hands-on CSV station lab without repeating that lab.

Essential questions

  1. What extra steps sit between 'clean CSV' and 'honest test score'?
  2. How can a brilliant feature be cheating?
  3. Why tune on dev and touch test once?

Objectives

  1. Card-sort the lifecycle into a single pipeline including feature engineering and three-way split.
  2. Identify at least three leakage patterns (future info, target in features, test peeking, duplicate rows across splits).
  3. State what each split is for: train fits, dev tunes, test reports.
  4. Write a methods note that a DA.3 potato-yield lab would still have to obey.

Key vocabulary

Feature engineering
Building new inputs from raw columns (GDD from temps, 'is_weekend' from a date, a ratio of wind to RH) before the model sees them.
Development / validation set
The middle split used to compare choices (features, thresholds, model family) without touching the test set.
Data leakage
The model sees information it would not have at prediction time, or sees test rows while training, so scores look fake-good.
Hyperparameter
A knob you set (tree depth, threshold) using dev performance, not a weight the training algorithm fits on its own.
Munging
Cleaning and reshaping raw data (types, missingness, duplicates) before features are built. Same word as in DA.3, methods view now.

Teacher background

DA.3 walked a potato-yield or wildfire CSV through collect → mung → features → split as a station lab and stopped before training. AIM.8 is the methods lecture: complete lifecycle, feature engineering as a craft, leakage as the central sin, three splits with roles, where training actually sits. Do not rerun the 40-row stations. Use a large pipeline poster and a leakage card-sort. Example features: GDD from min/max temp (procedural feature, AIM.4), county one-hot, lagging yesterday's cfs (careful: time). Leakage: using final fire acres to predict ignition morning; using the test mean to fill missing train values; shuffling so the same farm appears in train and test; peeking at test accuracy 40 times. Spreadsheet is only for a leakage demo (a 'future' column). Python is extension.

Materials and prep

Materials

  • Pipeline poster strips: collect, mung, feature engineering, split, train on train, tune on dev, evaluate once on test, deploy.
  • Leakage cards (8): future column, target encoded in an ID, test-mean imputation, duplicate rows, random split on time series, peeking 40 times, using post-treatment acres, student PII as a feature.
  • Legal-feature cards: GDD, yesterday's public cfs, county, wind×RH (engineered), planting month.
  • DA.3 reminder slip: 'You already split a CSV. Today we add engineering and leakage.'

Before class

  • If DA.3 has not been taught, give a 3-minute summary; do not convert this period into the station lab.
  • Print a bad pipeline that trains on all rows then 'tests' on a subset of them.

Instructional sequence

Where does training sit?

5 min
  1. Students place a TRAIN magnet on a blank timeline. Most will put it too early.
  2. Reveal: after collect, mung, engineer, split — and only on the train slice.

The complete procedure

12 min
  1. Walk the standards list: collecting, munging, feature engineering, train/dev/test splits. Add the methods extras: train, tune on dev, test once.
  2. Feature engineering examples: GDD; a frost binary from temp; a 3-day wind mean. Each is a choice that can leak if it uses the future.
  3. Three splits: train (fit), dev/validation (choose features, depth, threshold), test (report). Peeking is leakage.
  4. Show one future-column leak on the wildfire story from DA.3 without reopening the full CSV kit.

Build the poster

10 min
  1. Class hangs pipeline strips in order. Debate whether mung comes before or after split (fit mung stats on train only — advanced but worth saying).
  2. Place two legal features and two leaks on the poster.
  3. Rewrite the bad pipeline (train on all rows) as a crime scene.

Leakage card-sort

10 min
  1. Pairs sort eight cards into leak / not leak / depends, with a because-sentence.
  2. They write where in the pipeline they would catch each leak.
  3. Offline is the sort itself.

Real-world examples

  • Energy load: using tomorrow's actual MW as a feature to 'predict' tomorrow is leakage; using a weather forecast is not (if that forecast would have existed).
  • Yield: post-harvest price as a feature for a planting-time model is future info.
  • Sports: using the final score to predict in-game win probability at halftime is leakage; using score at halftime is legal.
  • River gauges: imputing missing train cfs with the test week's mean smuggles test information.

Hands-on activity

Methods note for the DA.3 table

8 min
  1. Without redoing stations, students write a half-page methods note: how they would engineer two features, how they would split in time vs. random, how they would keep test sealed while tuning.
  2. Swap notes; highlight any leak.

Discussion questions

  1. Should feature engineering happen before or after the split, and what goes wrong either way?
  2. If we only have 40 rows (DA.3), is a three-way split still right? (Maybe nested or a tiny dev — methods honesty still matters.)
  3. Why is 'we shuffled and got 99%' a confession, not a brag, on a time series?
  4. How is leakage different from the DA.5 coverage gap? (Cheating vs. missing people.)

Differentiation

Support

  • Pipeline strips numbered; sort only leak vs. not leak on four cards.
  • Sentence stems for the methods note.

Challenge

  • Explain target leakage vs. train-test contamination as two species.
  • Design a time-based split for wildfire days that still keeps counties represented.

Multilingual learners

  • Pipeline verbs with glosses; card-sort is visual.
  • Methods note allowed bilingual with English stage names.

IEP / 504

  • Poster placement as the artifact; skip the half-page if writing is the barrier.
  • Fewer leakage cards.

Assessment

Formative

  • TRAIN magnet placement after instruction.
  • Two leaks correctly placed on the poster.

Summative

  • Sorted leakage deck with because-sentences plus a methods note that includes feature engineering and a sealed test.
  • Repeating DA.3 station work without engineering/leakage does not meet AIM.8.

Success criteria

  • Student lists collect, mung, feature engineering, and train/dev/test in a sensible order.
  • Student explains at least two leakage types.
  • Student can say what dev is for.

Responsible use, ethics, and privacy

Responsible use

Do not scrape new student data to 'practice collection.' Collection here is a pipeline stage, not a survey of the class.

Ethics

A leaked test score can ship a harmful model with a straight face. Honesty in the procedure is an ethical duty.

Privacy

Leakage card 'student PII as a feature' is a hard no under FERPA, not only a methods fail. Public tables only.

Reflection

  1. Which leakage type would have fooled you last month?
  2. How will you remember that dev is not test?
  3. What is one feature you want to engineer on the potato table that is legal at planting time?

Homework

Diagram the lifecycle on one page, including feature engineering and three splits. Add three leakage examples (one time-based, one future column, one peeking). Do not redo the DA.3 CSV.

Closing

DA.3 put a CSV through collect, mung, features, split. AIM.8 added the craft: engineering, leakage, and why three piles exist. Training is one box in a longer honest procedure. Last methods lesson: the score people quote is often a trap.

Extensions and cross-curricular links

Go further

  • 90-minute block: a longer leakage museum and a 'fit scaler on train only' spreadsheet demo; still not a second DA.3 station rotation.
  • Python extension: demonstrate leakage by including a future column and watching accuracy jump — then drop it.
  • Guest: a data engineer (Micron, lab, county) on munging vs. engineering in their pipeline.
  • Printed pipeline tracing for subs.
Statistics
Holdout and cross-validation are the cousins of train/dev/test; peeking is p-hacking.
History / social science
Using later knowledge to explain an earlier event is the same intellectual leak.
Agriculture
Features known at planting vs. at harvest decide whether a yield model is a forecast or a diary.