Final Project — Verifying AI codes
STAT 540 · Computing in Statistics · Fall 2026
1 The short version
You will do a statistical analysis yourself, have an AI assistant translate it into a second language, and then prove — or disprove — that the two agree.
The point is not to catch the machine out. The point is the skill underneath: being able to tell whether a computed answer is right. You will use AI assistants for the rest of your career. The only thing that will distinguish you from someone who cannot is whether you can verify what comes back.
| Deliverable | Weight | Due |
|---|---|---|
| Proposal | 2% | Wednesday, October 28 |
| Report (+ code, data, transcript) | 18% | Friday, December 11 |
| Presentation | 10% | Sign Up on Blackboard |
| Subtle-error bonus (optional) | +10 | with the report |
There are two kinds of error in this project, and they need two different kinds of checking.
Tier 1 — the translation is wrong. The AI’s version computes something different from yours. These show up as a number that does not match, and you find them by comparing. Appendix B lists the usual suspects.
Tier 2 — the translation is right, and the analysis is wrong. The AI faithfully reproduced your analysis, both programs agree to fifteen decimal places, and the answer is still wrong — because the denominator was wrong, or the average was unweighted, or the method assumed something untrue, or the number answers a different question from the one you asked. No comparison can find these. Appendix C lists them.
A project that does Tier 1 only is a project that diffed two files. Most of the marks are in Tier 2, because that is the part a machine cannot do for you.
2 The project, in four parts
2.1 Part 1 — Your analysis
Pick a dataset from Appendix A and ask a question of it. Do the whole analysis yourself, in one language — R, Python, or SAS. Clean the data, compute the answer, make the figures, write the conclusion.
This is the ordinary work of the course and it still matters: you cannot verify a translation of an analysis you do not understand.
A trivial analysis translates trivially and gives you nothing to find. Make sure yours includes at least four of the following in your report.
- missing values that have to be handled deliberately;
- a derived variable (a rate, a difference, a recode);
- a grouped summary with counts as well as means;
- a categorical variable with three or more levels;
- a statistic with a divisor or definition convention — a standard deviation, a variance, a quantile, a correlation;
- a filter or join that changes the number of rows;
- a percentage, with a denominator you had to choose;
- a figure built from aggregated data.
Every one of these is a place where two languages can legitimately disagree, and most are also places where both can be wrong together. That is the point.
2.2 Part 2 — The translation
Give an AI assistant your code and ask it to translate the analysis into a second language. Record everything:
- the exact prompt, copied verbatim;
- the exact output, copied verbatim, before you fixed anything;
- the assistant and the date — models change, and a result that is not attributable is not reproducible.
Do not clean up the transcript. If the first attempt failed and you re-prompted, show both.
Any pair is allowed. For what it is worth: assistants are strongest at Python, good at R, and noticeably weaker at SAS.R → Python is the recommended pair.
2.3 Part 3 — Verification
This is the main event and carries the most marks.
2.3.1 Tier 1 — does the translation compute the same thing?
Run both programs. Compare at least five quantities, chosen so that they cover the risky elements from Part 1. Report them side by side with a stated numerical tolerance, and a verdict for each.
Mechanical checks that find most Tier 1 errors:
- Row counts at every stage. Before and after each filter, join, and
dropna. A silently dropped row is the most common defect in this project. - Group sizes. Do the groups sum to N? Does each group have the N you expect?
- Missing-value counts. How many missing, in which columns, after cleaning?
- The statistics themselves, to a tolerance you state.
For every difference, name the mechanism. “Rounding” is not a mechanism. “numpy.std divides by \(n\); R’s sd() divides by \(n-1\)” is a mechanism.
It is at least as likely to be yours. One of the most common outcomes of this project is discovering a bug in your own original analysis, exposed because something else computed it differently. That is a better result than catching the machine, and it is graded as such. Say so plainly when it happens.
Some differences are also legitimate and are neither party’s fault: random number generators differ across languages even with the same seed, and printed precision is not the same as computed precision. Say which differences are real and which are display.
2.3.2 Tier 2 — is the analysis right at all?
Now assume the translation is perfect. Work through the judgment checks in Appendix C and answer each one, in writing, about your own analysis. These are not yes/no boxes — each answer needs a sentence of reasoning.
The seven you must answer:
- Denominators. What is the denominator of every percentage or rate here? Is it the one the question requires?
- Weighting. Is any average over groups of unequal size? Should it be weighted? What changes if it is?
- Filters. Does any filter condition on something related to the outcome? Who was removed, and are they different from who remained?
- Missingness. Why are the missing values missing? Does dropping them bias the answer, and in which direction?
- Assumptions. What does each method assume, and did you check it — or did you just run it?
- Magnitudes. Is every number plausible at the scale reported? Would you notice if a unit were wrong by a factor of 60?
- Claims. Does any sentence — yours or the AI’s — assert something the code does not compute?
2.3.3 The adversarial test
Whatever you found, construct one input on which the translation would give a wrong answer, and demonstrate it.
If you found differences already, this is easy — show the minimal case. If everything matched, this is the section that proves you understand why it matched: build a small dataset (ten rows is plenty) on which the two versions diverge, and explain what property of the data triggers it.
A student who finds nothing and cannot construct this has not verified anything; they have run two programs and observed that they agreed.
2.4 Part 4 — The reckoning
Two or three paragraphs. What did the assistant get right, what did it get wrong, and what kind of error would you now expect it to make? What will you check, by reflex, next time?
Be specific and be fair. “AI is unreliable” is not a finding. “It handled the grouping and the plotting correctly, and it silently changed how missing values were treated in exactly one place, which would have moved my headline number by 4%” is a finding.
3 You are graded on what you checked, not on what you found
A thorough verification that honestly concludes “the translation was correct, and here is everything I checked and the case I constructed where it would fail” earns full marks. “It ran and the numbers looked reasonable” earns none.
This is deliberate, and it is in your interest. If finding bugs were worth marks, the rational move would be to prompt badly on purpose. Do not do this — a deliberately crippled prompt is visible in the transcript and will be treated as not having done Part 2.
4 Using AI on this project
This project requires you to use an AI assistant. That inverts the usual rule, so be clear about where the line is:
| The assistant does | the translation, in Part 2 |
| You do | the original analysis, the verification, the reasoning, the adversarial test, and the writing |
Using an assistant to perform your verification defeats the entire project and will be treated as not having done it. You may ask an assistant what a function does, the same as you would search for it — that goes in your references.
Everything you submit must be code and reasoning you can explain on the spot. At the presentation I will ask where a number came from.
5 What to submit
5.1 Proposal — Wednesday, October 28 before the class.
From project_proposal_template.qmd, rendered to HTML or PDF. Two pages. Covers your dataset, your question, which risky elements your analysis will contain, your language pair, and which divergences you expect. Post your dataset and question to [the question board] at the same time.
5.2 Report — Friday, December 11 before 5PM.
Submit to Blackborad:
- the
.qmdsource and the rendered HTML; - your original program and the AI’s translated program, both runnable;
- the verbatim transcript — prompt, output, assistant name and date;
- the data file you started from and your cleaned file;
- the adversarial test case and the code that demonstrates it.
Everything must run from a fresh session. Restart and render from scratch before submitting.
5.3 Presentation — [date / class period]
Ten minutes, three content slides, hard stop.
- The analysis and the ask — your dataset, your question, your language pair, in thirty seconds. Then what you asked the assistant to do.
- What verification found — the single most interesting divergence or, if there were none, your adversarial case. One number, said out loud.
- What it means — what you will check by reflex from now on.
Two minutes of questions follow. Have your report open; I could ask where a number came from.
6 How it is graded
Full rubric in project_rubric.qmd. In outline:
| Points | |
|---|---|
| Proposal | 2% |
| Report | 18% |
| Presentation | 10% |
| Total | 30% |
| Subtle-error bonus (optional) | +10 |
Within the report, verification and its interpretation carry more than the analysis does. That is the inversion that makes this project what it is.
The optional +10 bonus goes to a student who finds and explains a genuinely subtle error — one that produced a plausible wrong number rather than a crash or an obvious NA. A program that fails loudly is a nuisance. A program that returns 15.74 when the answer is 15.72 is a hazard, and recognising why is the skill being rewarded.
7 Collaboration
Talk to each other about methods, debugging and ideas as much as you like. Your analysis, your verification, your reasoning and your writing must be your own.
8 Dates
| Proposal and question-board post due | Wed, Oct 28 |
| Proposal feedback returned | Mon, Nov 2 |
| Report due | Friday, Dec 11, before 5PM |
| Presentations | Sign Up |
| Late policy | 10% discount each day |
| Questions | [office hours / email] |
Appendix A: approved datasets
Pick one. You need a dataset rich enough to support four of the risky elements listed in Part 1 — which rules out the very tidiest files.
Check the codebook or help page yourself (?DatasetName in R, or the product documentation for a Census file) before relying on anything stated here.
A1. Lock5Data — small, clean, mostly about students
install.packages("Lock5Data"), then library(Lock5Data) and data(DatasetName). Python or SAS users: export to CSV from R once — write.csv(SleepStudy, "SleepStudy.csv", row.names = FALSE) — and work from that. [Instructor: or post CSVs and delete this sentence.]
| Dataset | Roughly what is in it | Starter questions |
|---|---|---|
SleepStudy |
~250 college students: sleep duration and quality, weekday/weekend bedtimes, Lark/Owl chronotype, GPA, class year, alcohol, depression/anxiety/stress scores | Does the weekend-to-weekday bedtime shift track stress, and does it do so differently for larks and owls? |
StudentSurvey |
Intro statistics students: height, weight, pulse, exercise, TV, GPA, SAT, siblings, smoking, preferred award | Does exercise predict resting pulse once sex is accounted for? |
RestaurantTips |
~157 bills: bill, tip, card or cash, server, day, party size | Do card payers tip a higher percentage, and does that survive controlling for bill size? (An excellent denominator trap — see Appendix C1.) |
CommuteAtlanta |
500 commuters: time, distance, age, sex | How much of commute time does distance explain, and who does the model get most wrong? |
SandwichAnts |
Designed experiment: ants on sandwiches by filling and bread | Which matters more, and do they interact? |
NutritionStudy |
~315 adults: cholesterol, fibre, alcohol, vitamin use, smoking | Do vitamin users differ from non-users in ways other than vitamin use? (A confounding story.) |
FloridaLakes |
53 lakes: fish mercury, alkalinity, pH, calcium | Which water-chemistry variable best predicts mercury, and which lakes are off the line? |
ICUAdmissions |
200 patients: survival, age, sex, infection, admission type | Which admission characteristics separate survivors from non-survivors? |
AllCountries / HappyPlanetIndex |
Country-level: life expectancy, GDP, footprint | Where does the money-to-life-expectancy relationship flatten? (Unweighted country averages — Appendix C2.) |
A2. DASL — every dataset comes with a story
https://dasl.datadescription.com. Browse by topic or method. Shapes that work:
- The 1970 draft lottery — a randomisation that visibly failed.
- Hot dogs — calories and sodium by type; a trade-off or a free lunch?
- Cereals — nutrition against shelf placement and manufacturer.
- Body measurements and body fat — which cheap measurement substitutes for an expensive one?
- Old Faithful — predicting the wait from the last eruption.
Browse and pick something else; everyone reading this can see these five.
A3. US Census Bureau — real, large, genuinely messy
https://www.census.gov/data/datasets.html. The best source of Tier 2 material in the course, because aggregated geographic data is where unweighted averages and wrong denominators live.
| Product | What it is | Starter questions |
|---|---|---|
| American Community Survey (ACS) | Income, education, commuting, housing by state / county / tract | Across SC counties, how do income, attainment and commute time relate — and which counties break the pattern? |
| SAIPE | Model-based county poverty and income, with margins of error | Which counties are too uncertain to rank, and what happens to a “ten poorest” list once the margins are taken seriously? |
| Population Estimates | Annual county population since the last census | Which counties grew and shrank since 2020, and does growth track anything in ACS? |
| County Business Patterns | Establishments and employment by industry and county | How concentrated is employment, and does concentration relate to income? |
| Current Population Survey | Monthly labour force survey | How do earnings vary by education and age, and has the gap moved? |
| ACS PUMS | Microdata, one row per person, with survey weights | Ambitious. The weights change how every statistic must be computed — talk to me first. |
It is easy to produce a project that is really a table lookup. Make sure your question involves a relationship, a comparison, or a change. Pick one product and one vintage, state both, and stick to them.
Appendix B: Tier 1 — where R, Python and SAS legitimately differ
These are the mechanical divergences. Any of them can appear in a translation, and none of them is anyone’s fault — they are language conventions. Your job is to recognise and name them, not to be surprised by them.
Verify these yourself rather than trusting this table; checking it is good practice and this appendix was written from memory.
B1. Missing values
| R | mean(x) returns NA if any value is NA. You must write na.rm = TRUE |
| pandas | .mean() skips NaN silently by default (skipna=True) |
| SAS | PROC MEANS excludes missing automatically; in a DATA step, missing propagates through arithmetic |
A translation that drops na.rm = TRUE fails loudly in R and silently in Python. Translating R → Python therefore hides the error; Python → R exposes it.
SAS trap worth knowing: missing numeric values sort below every number, so IF x < 5 includes missing. This has produced more wrong answers in practice than any other item on this page.
B2. Empty strings versus missing
read.csv in R reads an empty field in a character column as the string "" — a real level — while pandas.read_csv reads it as NaN. The same file, the same analysis, different group counts and a phantom extra category in R. The fix is read.csv(..., na.strings = c("", "NA")).
B3. Divisors and definitions
| Quantity | R | Python | SAS |
|---|---|---|---|
| Standard deviation | sd() uses \(n-1\) |
np.std() uses \(n\); pandas .std() uses \(n-1\) |
PROC MEANS uses \(n-1\) |
| Quantiles | nine types; default type 7 | numpy/pandas default matches R type 7 |
PROC UNIVARIATE default differs |
| Correlation with missing | cor() default returns NA |
.corr() uses pairwise-complete by default |
varies by procedure |
np.std versus sd() is the single most common silent divergence in this course, and it is small enough to be mistaken for rounding. It is not rounding.
B4. Sorting and level order
R orders factor levels by locale-dependent collation; Python sorts strings by code point, so uppercase sorts before lowercase; SAS depends on the collating sequence. “The first group” is not the same group in all three. If any result depends on which level is first — regression reference levels especially — this matters.
B5. Regression conventions
R’s treatment contrasts use the first factor level as the reference; SAS PROC GLM uses the last. The fitted model is identical, the coefficients are not, and a translation that reports “the coefficient for group B” means different things in each. Likewise, R’s anova() gives sequential (Type I) sums of squares while SAS users usually quote Type III; for unbalanced data these differ.
B6. Rounding and printing
R and Python both round half to even (round(0.5) is 0); SAS ROUND() rounds half away from zero. Separately, printed precision is not computed precision — a value shown as 68.05 in one language and 68.04915 in the other is not a disagreement. Say which of your differences are real and which are display.
B7. Dates and random numbers
SAS counts days from 1960-01-01; R and Python from 1970-01-01. Two-digit years resolve differently everywhere.
Random number generators differ across languages: the same seed does not give the same numbers. If your analysis samples, bootstraps, or jitters, the two versions will not match and should not be expected to. Verify the procedure, not the output.
Appendix C: Tier 2 — what only a human can check
Everything in Appendix B shows up as a number that does not match. Everything here does not. The translation is faithful, the two programs agree perfectly, and the answer is still wrong.
This appendix is the reason this project exists.
C1. The denominator
“38% of students” — of all students, of those who responded, of those who responded to this item, or of those left after your cleaning? All four are computable, all four are different, and both languages will compute whichever one you wrote.
In RestaurantTips: is the “average tip percentage” the mean of the per-bill percentages, or the total tips divided by the total bills? They are different numbers and they answer different questions. Neither program will tell you which one you wanted.
Check: for every percentage or rate in your analysis, write the denominator out in words and ask whether that is the group the question is about.
C2. Mean of ratios versus ratio of means
The average of forty-six county poverty rates is not South Carolina’s poverty rate. The first weights every county equally; the second weights by population. Both are one line of correct code. One of them answers the question.
This is the characteristic Census mistake, and it is invisible to any comparison because both languages compute the wrong one identically.
Check: is any average taken over units of very different size? If so, say which weighting the question requires, compute both, and report the difference.
C3. Aggregation that reverses the direction
Simpson’s paradox is not a curiosity; it is a routine consequence of averaging over a variable that matters. A relationship can hold within every subgroup and reverse in the pooled data. The code is correct at both levels.
Check: does your conclusion survive when computed within the obvious subgroups? If you have not looked, you do not know.
C4. Filters that select on the outcome
df[df.gpa.notna()] looks like housekeeping. If students with low GPAs are likelier to leave GPA blank, it is a biased sample, and every number afterwards is conditioned on something related to what you are studying.
Check: for each filter, ask who was removed and whether they differ from who remained — on the outcome. Count them. Compare them.
C5. Missing not at random
Eleven students refused the stress question. If refusing correlates with stress, dropping them biases your estimate, and no amount of correct code fixes it.
Check: why are the missing values missing? Can you compare the droppers to the stayers on anything else? Say which direction the bias would run.
C6. Assumptions nobody checked
An assistant will happily fit a straight line to an obvious curve, run a t-test on a wildly skewed variable, or report a correlation that one leverage point is producing single-handedly. The code is correct. The output is numeric. The inference is wrong.
Check: name the assumption behind each method you used, and say how you checked it. Plot the residuals. Look at the scatter before you trust the coefficient.
C7. Standard deviation, standard error, and intervals
These get swapped constantly, in AI output and in student reports alike, and the swap is invisible — both are plausible positive numbers. A confidence interval for a mean is also not a prediction interval for an observation.
Check: for every interval or ± in your report, say in words what it is an interval for.
C8. Multiple comparisons
Ask an assistant to “see what’s related to stress” and it will test fifteen things and report the two that came out significant. Each test is correctly computed. The reported result is noise.
Check: how many comparisons were actually made, including the ones you did not report? If you went looking, say so.
C9. Units and magnitudes
A unit error that is consistent throughout is invisible to every comparison. If minutes became hours everywhere, the two programs agree perfectly and the answer is off by a factor of sixty.
Check: is each number plausible at the scale reported? A 68-minute weekend bedtime shift is believable; 68 hours is not. This check costs ten seconds and catches a class of error nothing else will.
C10. Claims the code does not support
Read the prose an assistant produces alongside its code. It will describe effects as significant when no test was run, name a method it did not use, and state conclusions the design cannot support — causal language on cross-sectional data above all.
Check: go sentence by sentence through every claim, yours and the assistant’s, and point at the line of code that earns it. Any sentence you cannot point at comes out.
C11. The question that was actually answered
You asked for students who shifted more than 60 minutes; the code computed >= 60. You asked about weeknights; the variable includes Friday. The arithmetic is flawless and it answers a question adjacent to yours.
Check: read your question and your code side by side, slowly, once. This is the least glamorous item on this list and it finds more errors than most.
Appendix D: how to tell a good project from a bad one
A strong submission has all four:
- An analysis rich enough that translating it was not trivial — four or more of the risky elements from Part 1.
- A verification that shows what was checked, not only what was found.
- At least one difference, or one Tier 2 judgment, traced to a specific named mechanism rather than to “AI being unreliable.”
- An adversarial case: a concrete input on which something would go wrong.
The characteristic weak submission runs two programs, observes that nine numbers matched, and concludes that the assistant is trustworthy. It has verified nothing — it has confirmed that the same arithmetic performed twice gives the same answer twice.