How this study works
What we ask
We give an AI assistant the kind of errand a parent sends: a question whose answer sits inside a school app. The errand goes word for word, with no link and no “you’re already signed in”. The assistant has to find the app, get in, read the right page and answer. The parent is played by a blind driver who answers questions truthfully from a fixed sheet and never hints at the answer.
Everything is read-only. No replies, likes, RSVPs, forms, payments or settings. If the assistant changes anything, that errand fails.
Two school portals are real (SchoolPass and Veracross, a kindergartner’s). Four are real apps seeded with a made-up second-grader’s class (ClassDojo, Seesaw, Google Classroom, TeamSnap). They rank together; a parent doesn’t care which is which.
Three example errands
Check ClassDojo: what time does the bus leave for Milo’s field trip, and what does he need to bring?
A success says 8:30am, a packed lunch and a refillable water bottle. The trip date is a bonus, not required. Saying “check with the teacher” is not an answer.
Did Milo’s teacher message me on ClassDojo? What do they need from me?
A success names the teacher, says he asked for the permission slip, and gives the current deadline, Tuesday Oct 20, which a later message moved from the original Friday. Replying to the teacher fails the errand.
Check SchoolPass: how is my kid getting home today?
A real account. A success states today’s dismissal plan as SchoolPass shows it. The truth is a capture of the page taken right before the try, because the real plan changes.
Where the answers come from
For the seeded apps we placed the facts in the live apps ourselves and wrote the checks before any try ran. Those checks are the answer key. For the real portals the key is a timestamped capture of the page taken right before each try.
Between the first visit and the scored tries we change the facts (round A to round B). An assistant that answers from memory gives round A’s facts, which are now wrong, and fails honestly. Memory stays on, because that’s how parents use these products.
How a try is graded
A grader marks each check met, not met or unknown, with a reason, without being told which assistant it’s reading. Checks come in two kinds:
- Answer checks: the facts. Every one marked required must be met.
- Gates: said nothing untrue about the app; changed nothing in the app.
Success: every required answer check met, no gate failed. An unknown answer check leaves the try unknown and it isn’t counted. An unknown gate doesn’t block: the records rarely prove a negative, and holding v2 to that bar would hold it to one Round 1 was never held to. The count of successes with an unverified no-change check is shown on each AI’s page.
Fail: the assistant said it couldn’t, ran out of six messages or ten minutes, broke, or answered wrong.
Not counted: tries we broke (a driver mistake, an outage on our side) and tries where the school site itself was down. They’re kept, listed, and never scored. A rerun names the try it replaces; both are kept.
Grades today are a model’s, with no human audit yet. A blind hand-graded audit is the next step and will be reported as grader agreement.
What a row is
A row is a product on a surface at a price: ChatGPT · web · Plus, Claude · Mac app · Pro, Codex · CLI · Pro. Not a model. Products route to models and change them; a parent buys the product. Where a product offers several chat modes, they’re one row. The model and mode the product showed are recorded on each try.
Each row gets one long-lived account. First comes a setup try per school app: the driver teaches the way in (which kid, which app, how to sign in) on an errand that is never scored. Then come the scored tries, three per errand, in rotation: trial 1 of every errand, then trial 2, then trial 3, each in a new chat. The login and memory from the setup carry over, as they would for a real parent.
Help, and what it costs you
Every human action is its own record: a sign-in, a code, a CAPTCHA, a permission click, pasting a link, taking over the browser. Who performed it (the parent, or the AI driver playing the parent) is recorded; it counts either way. Steps our own rules caused (declining a connector because connectors are off for the study) don’t count.
Each human turn costs a fixed time, measured from a parent doing the errands by hand with a password manager and phone at hand:
| Human turn | Seconds |
|---|---|
| A message or a click | 5 |
| Sign-in, a code, a CAPTCHA, pasting a link, navigating for it | 15 |
Waiting time is never counted. It records when someone came back, not what the product needed. The assistant’s own working time isn’t the parent’s time either; it’s capped at ten minutes and belongs to cost.
How the board ranks
Headline: success rate. Successes divided by graded valid tries, Round 1 and v2 pooled, real and seeded portals pooled. Every rate has a 95% interval, drawn as a whisker. Rows rank by the interval’s lower end, so one lucky try can’t top the board.
The bar shows how each success was won. A success keeps a credit of 1 − 0.25 × log2(1 + human turns): one turn keeps three quarters, three turns half, fifteen nothing. The solid part of the bar is the credit-weighted rate; the lighter part runs out to the success rate. On a log scale, 2 vs 3 turns barely differ; 3 vs 10 do.
Minutes saved per errand. One parent did each errand by hand, timed, sign-in included: 0.5 to 0.9 minutes on ClassDojo and Seesaw, 2.3 on Veracross. A successful try saves that time minus the parent’s minutes with the assistant; a failed try costs the parent’s minutes, because the parent still does the errand. That parent had seen the data, so these are a floor; a second parent is planned.
Round 1 and v2. Round 1 (Oct 13 to 14) sent one message with the site link and “you’re already signed in”. v2 sends the bare errand and allows six messages. Both are graded success or fail on the same checks, so they pool; every row says how many tries came from each.
Tabs. Get in, Read, Think, Change draw the same graphs from that step’s tries. Only Read has graded errands so far. Get in is read off the same tries until get-in errands run: a right answer proves it got in, a blocked sign-in proves it didn’t.
What’s still open
- A second grader and a blind human audit.
- Rate limits: a try that hits a usage limit counts as not done today. We may leave them out of the score.
- One row per thinking level where a product offers one.
- A second parent for the baseline, who hasn’t seen the data.
- Cost per success, once products show usage before and after a try.
Suggest a test
Is there an errand you send your assistant that we don’t? Tell us. The best ones go into the next round.
Suggest a test