Skip to content
Idaho AI Classroom
All lessons
AI Technical AptitudeData and Analysis9-12.AITA.DA.5

Better Examples, Fairer Model

9–12 CS / engineering50 minutes90 minutes (lab extension)

Standard quoted exactly

Investigate ways to improve the accuracy of a machine learning model and reduce bias by refining the quality of examples and nonexamples in the training data.

Clarification. Students don't actually need to train a classifier, rather they can look at datasets and determine what might need to be done to improve the model, by augmenting the data.

Example from the standards. A speech recognizer only recognizes adult, male, standard American English speakers because it was trained on news transcripts from the 1960s. Augmenting with women and children and with more varied accents would make the recognizer more robust.

Student-friendly learning targets

  • I can inspect a training set's examples and nonexamples and say who or what is missing.
  • I can propose augmentation (more examples, better nonexamples, more variety) without training a model.
  • I can explain how a narrow example set makes a model both less accurate and more biased.

Essential questions

  1. If the model never saw it, why would we expect it to recognize it?
  2. What is a nonexample, and why are bad nonexamples as dangerous as missing examples?
  3. When does adding more data help, and when does it just add more of the same bias?

Objectives

  1. Audit a small labeled set (speech, crop disease, or smoke photos described on cards) for coverage gaps.
  2. Separate the 1960s-news-speech story into: who is overrepresented, who is missing, what to augment.
  3. Write an augmentation plan: what to add, what nonexamples to add, what not to scrape.
  4. Connect accuracy and fairness: a model can be accurate on its narrow test slice and fail everyone else.

Key vocabulary

Example
A training item of the class we care about (a 'smoke' photo, a 'blight' leaf, a child's speech clip).
Nonexample
A training item of what the class is not (steam, dust, a healthy leaf). Weak nonexamples make false alarms.
Augmentation
Adding or transforming training items so the set covers more of the real world (more speakers, more lighting, more counties).
Coverage gap
A slice of the real world with few or no training items (children's voices; dryland farms; night-time imagery).
Robust
Still works when inputs vary in ways the designers should have expected: accent, age, camera, weather.

Teacher background

The clarification is the pedagogy: do not train. Students read datasets (card decks that stand in for public corpora) and prescribe augmentation. Teach the standards example plainly: a recognizer trained on 1960s broadcast news heard mostly adult men in a single accent, so it fails women, children, and other Englishes. Accuracy on that old news test set can look fine. Idaho analogs: a blight classifier trained only on Treasure Valley russets under noon sun; a smoke detector trained only on Boise foothills daylight; a river-ice model trained only on low-elevation gauges. Nonexamples matter: if 'not blight' is only perfect lab leaves, the model will panic on dust. Spreadsheet audit of label counts by group beats a GPU. No student voice recordings.

Materials and prep

Materials

  • Card deck A: 24 'training clips' described on paper (adult male broadcast, adult male broadcast, ... plus 2 women, 0 children, 0 Spanish-accented English).
  • Card deck B: crop-leaf labels — 18 russet blight, 2 healthy russet, 0 other varieties, 0 Palouse wheat, 0 night photos.
  • Card deck C: wildfire smoke vs. not-smoke — smoke at noon, steam from a mill, irrigation dust; missing night and high-elevation.
  • Augmentation plan template: add / transform / collect-ethically / do-not-collect.
  • Printed standards example quote for close reading.
  • Tally sheet by group (speaker, crop, lighting, county).

Before class

  • Print decks; the 'dataset' is the cards. Do not ask students to record their voices or photograph classmates.
  • Pre-count the imbalances so you can coach without spoiling the find.
  • Have the speech-example quote on the board for the last five minutes of direct instruction.

Instructional sequence

Who is in the room on these cards?

5 min
  1. Deal deck A face up. Students tally speaker type in 60 seconds.
  2. Ask: if this is the whole training set, whose words will the recognizer miss?
  3. Read the standard example aloud once, without unpacking yet.

Examples, nonexamples, augment

10 min
  1. Define example vs. nonexample with blight/healthy and smoke/steam.
  2. Walk the 1960s news story: overrepresented (adult male standard broadcast English), missing (women, children, varied accents), result (recognizer not robust), fix (augment, do not just collect more of the same men).
  3. Show a bad augment: 10,000 more identical russet noon photos. Quantity is not coverage.
  4. Show a bad nonexample set: 'not smoke' is only blue sky, so mill steam pages crews.

Audit deck B together

12 min
  1. Tally variety, disease, lighting, county on the crop deck.
  2. List coverage gaps: other varieties, dryland, cloudy days, healthy leaves with dust.
  3. Draft one augmentation sentence as a class: 'Add 30 labeled healthy and diseased leaves from Palouse wheat and Magic Valley sugar beet, including dusty nonexamples, from public research photos — not student phones.'

Prescribe for deck C (or A)

10 min
  1. Pairs pick smoke or speech remainder. Fill the plan template: three adds, one nonexample improvement, one ethical no (no classmate recordings, no private farms).
  2. They must say how they would check that the new items are labeled correctly (who verifies blight?).
  3. Offline is the default: cards and template, no training UI.

Real-world examples

  • Speech: the standards 1960s news story; Idaho 911 audio would fail the same way if trained only on adult dispatchers.
  • Crop disease: a U of I extension model trained on irrigated russets will misread dryland Palouse wheat fungus.
  • Wildfire smoke cameras in the Boise foothills at noon will treat mill steam in Lewiston or canal dust in the Magic Valley as fire — missing nonexamples.
  • River ice-on/ice-off trained only on Treasure Valley gauges will not represent high Sawtooth streams.

Hands-on activity

Before/after coverage table

8 min
  1. Students fill a two-column table: current counts by group vs. proposed counts after augmentation.
  2. They star any group that is still zero (a remaining fairness hole).
  3. Quick gallery: one gap per pair on the board.

Discussion questions

  1. Why can accuracy go up on the old test set while the model gets less fair?
  2. Is a synthetic (computer-drawn) blight leaf a legitimate augment? What could go wrong?
  3. Who should label the new nonexamples — a student intern or a plant pathologist?
  4. When should we refuse to collect a group even if it would help the model (children's voices at home, tribal land imagery)?

Differentiation

Support

  • Tally boxes pre-drawn; students only count and copy a sentence stem for the plan.
  • Use deck A only; it matches the written example most closely.

Challenge

  • Design a sampling quota (min 20% night, min three counties, min two varieties) and explain how it fights bias without claiming the model is 'unbiased.'
  • Critique a fake 'we added 1 million images' press release that does not change the mix.

Multilingual learners

  • Speech deck includes language/accent as a tally column; discuss that 'standard American English' is one variety, not the correct one.
  • Allow the augmentation plan in the student's strongest language with English dataset nouns.

IEP / 504

  • Fewer cards (12) with large labels; oral plan is accepted.
  • Avoid audio playback of 'bad recognizer' demos that can be frustrating; stay on paper descriptions.

Assessment

Formative

  • Warm-up tally of deck A.
  • Class augmentation sentence for deck B.

Summative

  • Completed plan template: gaps named, examples and nonexamples addressed, one ethical refusal, no claim that they trained a model.
  • A student who proposes 'just train longer' without changing the set has not met the standard.

Success criteria

  • Student distinguishes examples from nonexamples.
  • Student proposes augmentation that changes coverage, not just volume.
  • Student never uses classmate PII or private recordings as the fix.

Responsible use, ethics, and privacy

Responsible use

Do not record students to 'improve' a speech set. Do not scrape classmates' photos. Use described public corpora and teacher cards.

Ethics

Augmentation can become surveillance: collecting more of a group without consent is not fairness. Some gaps should be filled with public research sets or by declining the product.

Privacy

FERPA and basic dignity: no student voices, faces, or home farms. Children's speech in the example is a research-ethics topic, not a homework recording.

Reflection

  1. Which missing group in your deck would cause the most harm if ignored?
  2. How is a nonexample different from 'just more data'?
  3. What will you say the next time someone boasts about dataset size only?

Homework

Reread the printed speech-recognizer example. In five sentences: who is overrepresented, who is missing, what nonexamples might be missing, what you would add from public sources, and what you would refuse to collect.

Closing

The model is what it ate. Better examples and harder nonexamples beat a bigger pile of the same men, the same russets, the same noon photos. We still have not trained — we prescribed. Next we pick a model that actually fits the job, including XOR.

Extensions and cross-curricular links

Go further

  • 90-minute block: second deck plus a spreadsheet pivot of labels by county; write a one-page memo to an extension agent.
  • Python extension: plot class counts from a public CSV (no training); students still write the augmentation plan in prose.
  • Card-sort 'helps coverage' vs. 'more of the same' vs. 'unethical to collect'.
  • Connect forward to DA.7 (critique ImageNet) as the large-corpus version of this audit.
Health / CTE audio
Speech tech in clinics fails when training missed women, children, and accented English — the standards example is a patient-safety story.
Agriculture
Disease libraries must include the crops and counties actually farmed, not only the research station's favorite variety.
Civics
Who is missing from a government dataset is a representation question, not only a tech one.