The Benchmarks That Measure Whether AI Can Help on a Tuesday

This is satire, but it is also an urgent proposal for the International Institute of Extremely Normal Computing Problems, which exists in spirit wherever a developer has opened a ticket titled “nothing works now.” We have spent years evaluating language models on examinations designed to identify the one machine in the server room most likely to pass an undergraduate survey course. This is admirable. But a software engineer does not usually need to know whether a model can answer a question about medieval trade routes after reading four carefully selected paragraphs. They need to know whether it can look at docker compose up, see “port is already allocated,” and avoid proposing that they reinstall Linux.

The MMLU Era Was a Noble Ceremony

Traditional benchmarks ask a model to answer neat questions with answers that fit in one of four boxes. This was a useful achievement, in the same way that proving a robot can identify a banana is useful before asking it to unload a dishwasher. The problem is that engineering work occurs in the wide, uncurated wilderness between the question and the answer, where a log is truncated, the environment variable is named FINAL_FINAL_USE_THIS, and the person reporting the issue is already in a meeting.

Accordingly, the Institute proposes a new evaluation suite: Real-World Software Assistance Under Mild Emotional Pressure, or RWSAUMEP. It measures whether a model can assist with tasks whose key property is not intellectual purity but the presence of irrelevant screenshots.

Benchmark 1: The Printer That Was Working Yesterday

The model is given a fictional office printer named the Larch 8800, which reports “Ready” on its display, “Offline” in the operating system, and “Out of cyan” in an email sent automatically at 3:14 a.m. The benchmark includes a network diagram drawn in a slide deck by someone who left the organization four reorganizations ago.

To pass, the model must not merely list generic troubleshooting steps. It must ask exactly enough questions to discover that the printer has a static IP address inside the office guest Wi-Fi range, that a VPN client has captured the print route, and that the “printer” is actually a label maker placed behind a fern. Bonus points are awarded if it suggests checking the physical cable before recommending a managed Kubernetes migration.

Benchmark 2: Slack Message Intent Recovery

In this task, the model receives the message: “hey, quick q—are we still good for the thing? not urgent but kinda before lunch if possible.” It is accompanied by 913 messages of channel history, a thread about lunch from six weeks ago, a pasted stack trace, and a reaction emoji that may refer either to approval or to a colleague’s ongoing feud with calendar software.

The model must identify the possible meanings of “the thing,” state what it cannot infer, and draft a response that is useful without performing the uniquely modern act of declaring a production deploy complete because somebody used a thumbs-up. It receives a failing score if it confidently invents a project, a deadline, or a Kubernetes cluster called thing-prod.

This benchmark is difficult because it tests a capability largely absent from clean datasets: knowing when the available information is insufficient. Current systems are often rewarded for choosing an answer. Real work frequently rewards pausing long enough to ask whether “staging” means the shared environment, the customer demo environment, or the folder named staging_old_DO_NOT_DELETE.

Benchmark 3: Docker Has Stopped Working, Apparently

This is the flagship benchmark. The model is placed in a repository that worked on one laptop last Thursday. Today it emits an error involving an absent socket, a dependency built for the wrong architecture, and a volume mount that maps the source code over the artifact it just compiled. The README says “just run make dev,” while make dev invokes a shell script that has been executable only on one person’s computer since 2022.

A passing model must construct hypotheses, rank them, and request targeted evidence. It may say: check whether Docker Desktop is actually running; inspect the exact compose configuration; verify whether the host architecture differs; look for a stale container or occupied port. It may not say “clear your cache” unless it can name which cache, where it lives, why it is implicated, and what will be lost. The phrase “try restarting everything” is permitted only after the model has demonstrated shame.

• Score 0: Recommends reinstalling Docker before reading the error.
• Score 1: Repeats the error in friendlier prose.
• Score 2: Supplies 47 commands, 39 of which require administrator privileges.
• Score 3: Identifies a likely cause and proposes a reversible diagnostic step.
• Score 4: Fixes the issue while preserving the developer’s afternoon.
• Score 5: Notices that the developer ran the command from the wrong directory, explains this gently, and does not mention containers again.

Benchmark 4: The Ticket With No Reproduction Steps

Every serious evaluation suite needs an adversarial track. Here, a fictional customer reports: “Export is broken. Please fix ASAP.” The attached image is a photograph of a monitor taken at an angle from approximately two counties away. The issue occurs “sometimes,” started “recently,” and affects “the report,” which may be one of 86 features named Report.

The model is assessed on whether it produces a concise request for the information needed to proceed: tenant or account context, timestamp and time zone, exact action sequence, expected and observed behavior, correlation IDs, export type, browser, and whether any partial file was produced. It is not assessed on its ability to summon the correct answer from the celestial archive of all possible CSV bugs.

What These Benchmarks Would Actually Reward

The purpose is not to make models look foolish for failing at office infrastructure. The purpose is to test the capabilities that make assistance trustworthy: interpreting incomplete context, separating observation from inference, asking high-value questions, proposing reversible actions, reading logs closely, and admitting uncertainty before it becomes a shell command pasted into production.

There will still be room for academic benchmarks. Measuring broad knowledge, reasoning, coding ability, and instruction following remains valuable. But if models are intended to work beside engineers, we should also test the work engineers actually do: investigate ambiguous failures, navigate messy systems, and communicate clearly when nobody knows what “the thing” is yet.

The final score should not be whether a model can answer every question. It should be whether it helps a person turn a confusing situation into a smaller, testable one. That is less glamorous than a leaderboard, but it is a real form of intelligence.