How the playground is checked

This card describes the evaluation harness at /card-table/llm-eval. It evaluates a general language model on one narrow task with a known right answer. It is not a model card for any model, and it makes no claim about how a model behaves outside this task.

The question

COMP10001-Go's bonus question asked for every highest-scoring way to split a hand of up to ten cards into groups. The exact solver answers it by checking every set partition (115,975 for ten cards). The question here is how often a language model, given the rules and a hand, returns a grouping that is legal, and how often it is the best one.

Players

Inputs

• Hands are dealt by the same seeded dealer as the Card Table: hand 1 of a set is deal(seed), hand 2 is deal(seed + 1), and so on. The default set is 30 hands of 10 cards from seed 2019; the page offers 10 to 50 hands of 6 to 10 cards and any seed. The seed is shown next to every hand. The default was 10 hands until DR-005, which explains the change.
• Every player sees exactly the same hands, so comparisons are paired.

What the model is given

• A system prompt with the scoring rules in my own words (the task text is not reproduced), the card notation and the instruction to use every card exactly once.
• One hand per request, and a JSON schema whose card list is that hand. The reply must be {"groups": [[...], ...], "claimed_score": n}.
• Claude Haiku 4.5 (the default) runs at temperature 0 with up to 1,024 output tokens. Claude Sonnet 5.5 always thinks before answering; it runs at the effort the visitor picks (low by default) with up to 8,192 output tokens, because thinking counts against the limit. OpenAI models run at temperature 0 unless they are reasoning models, which do not accept it.
• Hands are sent one at a time, and nothing from one hand is shown to the model for the next.

How an answer is scored

• Valid: the reply matches the schema, uses every card in the hand exactly once, and every group of two or more cards is legal under the rules.
• Optimal: valid, and it scores the exact solver's best total.
• Score with fallback: the answer's score if valid; otherwise the hand played as singletons, which is what a careful player falls back to after rejecting an answer.
• Gap: best score minus the score with fallback. It is never negative, and a failed answer counts against it instead of being dropped.
• Claimed total: among valid answers, whether the model's own claimed_score equals the real score. A model's account of its own work is checked, not trusted.
• Outcomes: answered (then judged), malformed reply, cut off at the token limit, refused, or failed call. A failed call (rate limit after retries, network) is not the model's answer, so it is left out of every rate and reported separately. When calls fail, every player on the scoreboard is scored on the hands the model was scored on, so the rows stay comparable; the export also has the baselines on the full set.
• Latency: the response time of the final attempt. Retries and back-off waits are left out of the evaluation's latency figures; the audit log records both.

Uncertainty

• Validity and optimality rates carry Wilson score 95% intervals, which behave well for small samples and for rates near 0% or 100%.
• Mean gaps carry percentile bootstrap intervals, labelled nominal 95%: 10,000 resamples of hands, seed 10001. They run narrow at these sample sizes (see the coverage table below). When every hand in a sample has the same gap, the resamples cannot vary, so the page says "no interval" instead of printing a zero-width one.
• The LLM and the greedy baseline are compared hand by hand: the mean of (LLM score minus greedy score) with a paired bootstrap interval, plus how many hands each was ahead on.
• The statistics helpers are tested against statsmodels, SciPy and NumPy, and the Wilson intervals also against R's prop.test.

How often the intervals hold the truth

"95%" is a property of a method in the long run, and at small samples it can fall short. I checked both interval methods on the kind of data they are used for here, with cd web && pnpm sim:coverage (web/scripts/interval-coverage.sim.ts). The population is 3,000 hands of 10 cards dealt by the same dealer (seeds 100,000 to 102,999, apart from the evaluation's own seeds), solved exactly. The greedy baseline stands in for a model, because its gaps have the awkward shape a good model's would: 71% of hands have a gap of 0 and the rest spread out as far as 101 points. Each bootstrap cell is 4,000 simulated evaluations, so it carries about ±1.5 percentage points of Monte Carlo error. The Wilson column is exact.

The Wilson intervals hold their level. The bootstrap intervals for a mean gap do not: with 10 hands of zero-heavy gaps, a "95%" interval missed the true mean about one time in five, and in 3% of samples every gap was 0, so the interval collapsed to a point. That is why the default is now 30 hands, 5 hands is no longer offered, and the page labels the gap intervals "nominal 95%". Even at 30 hands they are somewhat too narrow, and the scoreboard says so.

Baselines on the default set

On the default 30 hands (seed 2019, 10 cards each), the greedy baseline is always valid and is optimal on 73% (56% to 86%) of hands. Its mean gap to the best score is 5.6 points (1.8 to 10.6). Playing every card alone is never optimal, 0% (0% to 11%), with a mean gap of 99.2 points (81.4 to 116.1).

On the first 10 of these hands, the old default, the same greedy baseline was optimal on 6, an interval of 31% to 83%. Thirty hands narrow that to 56% to 86%: enough to tell a strong player from a weak one, still not enough to rank two close ones.

What the judge can tell apart

• Cards left out, used twice, or not in the hand at all.
• Groups that break the rules: a run whose colours do not alternate, an Ace at the end of a run instead of inside it, a run with fewer than two non-Ace cards, or Aces as a set.
• A legal grouping that is not the best one, and by how many points.
• A claimed total that does not match the real score of the grouping.

Assumptions

• The rules engine is right. It is checked against the 2022 reference notebooks (500 groups, 200 groupings, 18 best-partition searches) and by property-based tests against an independent exhaustive enumeration.
• Hands dealt from a shuffled deck are a fair test. They are not weighted towards hard hands; many random hands have little to group, which favours the greedy baseline.
• One request per hand, with no retries on a wrong answer, is the setting being measured.

Limitations

• Each hand is asked once, so run-to-run variation is not measured.
• The gap intervals are percentile bootstrap intervals on skewed data, and in the coverage simulation they held the true mean less often than their nominal 95% (91% at 30 hands). The greedy baseline's gaps stand in for a model's; a model whose gaps are shaped differently would get different coverage.
• One prompt is used. A different wording could change the results; that is not measured.
• Results are only as recent as the run: providers update models behind the same id.
• Nothing here measures anything other than this card game; it says little about a model in general.

Audit and review

Every call is written to the AI audit log in the visitor's browser (/ai-log) with the full prompt, the JSON schema and generation settings (temperature, output-token limit, thinking effort), the raw and parsed reply, the validation result, the stop reason and any refusal category, latency (final attempt and total), token usage, and the visitor's accept or reject decision with an optional note. Results can be exported as CSV or JSON from the evaluation page, and the log from /ai-log; the evaluation's export reads decisions from the log, so a decision changed later is exported as it stands.

This statement says where AI is used on the COMP10001 Playground, what it never does, what data leaves the visitor's browser, and where a person stays in charge. It is informed by the Australian Government's Policy for the responsible use of AI in government (Digital Transformation Agency), the transparency principles of the EU AI Act, and the NIST AI Risk Management Framework. This is a personal portfolio project: it is not assessed against any of them and makes no claim of compliance.

What AI does here

There is one AI feature, and it is optional: the evaluation "Can an LLM find the best partition?" on the Card Table (/card-table/llm-eval). When a visitor adds their own API key and starts a run, a language model is asked to group each seeded hand of cards, and its answers are scored against the exact solver.

What AI never does

• It never runs without the visitor's own key and an explicit click on "Run".
• It never changes the original 2019 code, the rebuilt functions or their results. The counting, the cave searches, the scoring and the exact solver are deterministic code.
• It never scores itself or anyone else. Every answer is judged by the same rules engine as the rest of the site.
• It never answers on behalf of another model. If the provider declines, the hand is recorded as refused; there is no silent fallback.
• It never writes the text of this site at runtime.

Data sent to the provider

• The scoring rules (a fixed system prompt), one hand of card codes such as 8S 8C 9D, and a JSON schema. No personal data is put in the request. Like any web request, the call reveals the visitor's IP address, browser user agent and this site's origin to the provider; cookies and the referring page are not sent.
• The visitor's API key, in a request header, to the provider they chose: Anthropic (api.anthropic.com) or OpenAI (api.openai.com). Requests go directly from the browser. This site has no AI server code and never receives the key.
• The provider handles requests under its own terms and retention policy.

Keys and storage

• The key is kept in the browser's sessionStorage and is gone when the tab closes, unless the visitor turns on "remember on this device" (localStorage). "Forget keys" removes it.
• Any script on the page could read a stored key, including browser extensions. The site loads no third-party scripts, and its Content-Security-Policy only lets the page connect to itself and the two provider APIs, so a script injected into the page could not send the key anywhere else with fetch (DR-006). Browser extensions are not bound by the policy, so visitors should still use a key with a spending limit.

Human in the loop

• The visitor starts every run, can stop it at any time, and in-flight calls are cancelled if they leave the page.
• Each answer is shown with an "AI-generated" label next to the exact answer and the greedy baseline's answer. The label marks content a model returned; failed calls and refusals are shown as such, without it. The visitor can accept or reject each answer, and add a note saying why; the decision and note are stored with the call.

Transparency and records

• Every call is written to an audit log in the visitor's own browser (IndexedDB), with the prompt, the JSON schema and generation settings, the reply, the validation result, any refusal, the model, the latency, the token usage and the human decision. It is never uploaded. It can be viewed, filtered, exported as JSON or CSV, and cleared at /ai-log.
• Scores are reported with intervals and the number of hands, and the evaluation design is documented on /methods, including how often the "95%" intervals actually held the truth in a simulation.
• The recordings on the guided tour (/tour) and in the README show the evaluation running on mocked answers: the tour enters no key, and each provider call is answered inside the test browser by a fixed rule. Those frames are labelled "Mocked AI response for illustration", and their scores say nothing about any real model.

How this site was built

The 2019 Project 2 file was written by me, without AI tools. The 2026 revival, including this upgrade, was written with an AI coding assistant under my direction and review, and the commits record that.