Skip to content
Idaho AI Classroom
All lessons
AI Technical AptitudeAI Methods9-12.AITA.AIM.9

Accuracy Is a Trap

9–12 CS / engineering50 minutes90 minutes (lab extension)

Standard quoted exactly

Interpret AI results using common metrics and be able to communicate those to a diverse audience.

Example from the standards. A credit card fraud classifier is 99.9% correct using an accuracy metric, but that's only because 99.9% of transactions are not fraudulent.

Student-friendly learning targets

  • I can explain why accuracy looks great on a rare-event problem and still fails the job.
  • I can compute accuracy, precision, and recall from a confusion matrix and say them in ordinary language.
  • I can communicate the same result to a fire chief, a county commissioner, and a neighbor without hiding false pages.

Essential questions

  1. If a model is 99.9% accurate, what might it still be refusing to find?
  2. What does a false positive cost a fire crew, and what does a false negative cost a hillside?
  3. Which metric should you lead with for which audience?

Objectives

  1. Build a confusion matrix for a rare-event task (fraud or Idaho wildfire starts).
  2. Compute accuracy, precision, recall; identify false positives that page crews and false negatives that miss ignitions.
  3. Rewrite a 99.9% accuracy boast using the standards fraud example and a matched wildfire story.
  4. Produce three audience-specific sentences that stay true to the same numbers.

Key vocabulary

Accuracy
Share of all cases the model got right: (TP + TN) / all. Looks huge when the rare class is tiny.
Class imbalance
One class is much rarer than the other (fraud, fire starts, equipment failure). Accuracy then tracks the majority class.
Precision
Of the alarms we rang, how many were real: TP / (TP + FP). Low precision means crews get paged for camp stoves.
Recall
Of the real events, how many we caught: TP / (TP + FN). Low recall means ignitions we never saw.
Confusion matrix
A four-cell table of truth vs. prediction: true positive, false positive, true negative, false negative.

Teacher background

This lesson must teach class imbalance. Use the standards fraud example and an Idaho wildfire rare-event table so students feel both. Numbers that work on paper: 10,000 grid cells, 10 real starts. A no-fire-always model is 99.9% accurate and 0% recall. A jumpy model that flags 200 cells with 8 real fires: compute precision 8/200 = 4%, recall 8/10 = 80%, accuracy still high because of all the true negatives. False positives page crews (overtime, ignored future tones). False negatives miss a start (acres). Communication: fire chief wants recall and FP load; commissioner wants cost; public wants 'will it miss my canyon?'; engineer wants the matrix. Spreadsheet four cells plus formulas. Offline: printed matrix with chips to count. Do not hide the algebra; it is four numbers.

Materials and prep

Materials

  • Fraud one-pager matching the standard: 999,000 legit of 1,000,000; a dummy 'always legit' model at 99.9% accuracy.
  • Wildfire matrix kit: 10,000 cells, 10 starts; two model result cards (always-no vs. jumpy pager).
  • Four colored chips or paper squares: TP, FP, TN, FN.
  • Audience cards: fire chief, county commissioner, news reporter, model engineer, canyon homeowner.
  • Calculator or spreadsheet; printed formula strip.

Before class

  • Precompute both scenarios so you can coach arithmetic without scooping the insight.
  • If AIM.7 used paging, hang that flowchart and add metrics to the same gate.
  • Avoid graphic burn photos; the matrix is the visceral object.

Instructional sequence

99.9% — hire this model?

5 min
  1. Read the fraud example. Vote hire / do not hire.
  2. Reveal: the model predicted 'not fraud' every time. Accuracy is a trap when the event is rare.

Four cells, three metrics, one imbalance

12 min
  1. Draw the matrix. Name TP (caught fire/fraud), FP (false page), FN (missed event), TN (correct quiet).
  2. Accuracy = (TP+TN)/all. Precision = TP/(TP+FP). Recall = TP/(TP+FN).
  3. Work the always-no wildfire model: TP=0, FN=10, FP=0, TN=9990, accuracy=99.9%, precision undefined or 0, recall=0.
  4. Say costs in Idaho English: FP pages crews; FN burns acres. Neither is 'accuracy.'

The jumpy pager

10 min
  1. Give jumpy-model counts (e.g., TP=8, FP=192, FN=2, TN=9798). Compute the three metrics as a class.
  2. Ask which metric you would quote to a fire chief (recall + FP count) vs. a headline (not 97.9% accuracy).
  3. Connect to AIM.7: a human dispatcher is a safeguard against a low-precision flood.

Three audiences, one matrix

10 min
  1. Pairs write three communications (2–3 sentences each) for three audience cards using the same jumpy-model numbers. No new math that contradicts the matrix.
  2. They must mention class imbalance once in student language ('fires are rare, so…').
  3. Offline: chips on a printed matrix, then sentences.

Real-world examples

  • Credit card fraud: the standards 99.9% story; banks actually optimize precision/recall and dollar loss, not accuracy.
  • Idaho wildfire starts on a 10,000-cell grid: always-no looks like a genius on accuracy and a disaster on recall.
  • Hospital lab 'always healthy' classifiers on rare conditions — same trap, different harm.
  • Sports: a model that always predicts the favorite in a lopsided conference can be 'accurate' and useless for an underdog bet — class imbalance in outcomes.

Hands-on activity

Chip the matrix, then translate

8 min
  1. Students place chips for a third scenario (teacher choice: 10 frauds in 10,000; model catches 6, pages 30).
  2. Compute metrics. Peer-check arithmetic.
  3. Read one audience sentence aloud; class flags hidden accuracy boasts.

Discussion questions

  1. Is a 4% precision smoke alarm ever acceptable if recall is 80%? Who pays for the 96% false pages?
  2. Why is 'accuracy 97.9%' a dishonest lede for the jumpy pager?
  3. How would you explain undefined precision when TP+FP=0?
  4. Which metric should a canyon homeowner hear first?

Differentiation

Support

  • Formula strip and a matrix with blanks only for TP/FP; TN pre-filled.
  • Two audiences instead of three; sentence stems.

Challenge

  • Introduce F1 as the harmonic mean in one paragraph, still leading with precision/recall.
  • Show how moving a threshold trades precision vs. recall on a sketched curve (no software).

Multilingual learners

  • Metric names + 'alarms that were real' / 'fires we caught' glosses.
  • Audience sentences in the student's strongest language plus English for the commissioner.

IEP / 504

  • Chips and a calculator; one audience sentence.
  • Large matrix; skip extra scenario.

Assessment

Formative

  • Hire/do-not-hire vote plus the always-no reveal.
  • Correct jumpy-model arithmetic in guided practice.

Summative

  • Completed matrix with three metrics and three audience communications that do not hide FPs/FNs.
  • A student who only reports accuracy has not met the standard.

Success criteria

  • Student can define class imbalance with fraud or wildfire numbers.
  • Student computes precision and recall from four cells.
  • Student changes language for audience without changing facts.

Responsible use, ethics, and privacy

Responsible use

Do not use student discipline or attendance as a 'rare event' toy. Fraud (public-style numbers) and wildfire grids only.

Ethics

Choosing a metric chooses who pays: crews (FP) or landscapes and families (FN). That is a values decision reported as math.

Privacy

No real card-transaction files, no student financial data, no household addresses on the wildfire grid. Abstract cells and public-style counts.

Reflection

  1. When will you refuse to quote accuracy?
  2. Which audience was hardest to write for, and why?
  3. How does this change what you demand from a vendor demo?

Homework

On the take-home matrix (wildfire or fraud), compute accuracy, precision, and recall. Write three audience paragraphs (chief, commissioner, neighbor). Circle every time you were tempted to lead with accuracy and did not.

Closing

99.9% can mean 'we never tried to find the rare thing.' Precision asks whether pages were real; recall asks whether fires were caught; false positives tire crews. Quote the metric that matches the harm, in language the audience can use. That is the methods capstone.

Extensions and cross-curricular links

Go further

  • 90-minute block: threshold workshop — two cutoffs, two matrices, a precision-recall trade; write a dispatcher SOP.
  • Python extension: sklearn classification_report on a public imbalanced CSV; still require the prose translation.
  • Card-sort headlines into 'honest' vs. 'accuracy trap.'
  • Printed chip matrix for a no-device day.
Statistics
Sensitivity/specificity in biology and health are cousins of recall and true-negative rate.
Journalism
Lead with the number that answers the audience's question, not the one that looks biggest.
Civics / emergency management
False pages are a public-resource problem; metrics are a civic communication duty.