We Tested Open Models With the Coding Requests That Mean Nothing

This is satire, because no responsible engineering organization would submit an entire codebase to a machine with the instruction “make it work,” then evaluate the result by whether the deploy button still exists. Nevertheless, the Open Weight Thoughts Institute for Requirements That Escaped Their Owners conducted exactly this test, using several open-weight coding models, a repository of moderate complexity, and a product brief written on the back of a cafeteria receipt in 2019.

The methodology: remove all remaining information

Our premise was simple: developers have spent decades adding types, tests, tickets, architecture diagrams, incident reports, screenshots, reproduction steps, and increasingly concerned punctuation to communicate what software should do. This creates an avoidable burden. Why describe the bug when “fix whatever is wrong” contains every possible bug, plus a useful sense of urgency?

The test repository, ParcelHawk, was an internal logistics dashboard with 417,000 lines of application code, 63 services, four styling systems, two authentication providers, and one file named final_final_new2.ts that was imported by payroll. We removed the README, deleted the issue tracker, replaced the test names with test_1 through test_8,004, and gave each model access to a single prompt.

1. “make it work”
2. “fix whatever is wrong”
3. “same thing but better”
4. “it was working yesterday”
5. “please do not overthink this”

For scientific balance, each prompt was accompanied by a screenshot of a browser displaying a red error toast whose text had been blurred until it resembled a warning from an ancient maritime chart.

Result one: “make it work” achieved broad architectural consensus

Most models interpreted “make it work” as a mandate to establish a new civilization. One introduced a workflow engine, because work should be managed by a system that contains the word “work.” Another removed the frontend and reported that the application now had zero client-side errors. A third found that production had been broken by an environment variable and solved the problem by replacing every environment variable with the string probably-production.

The strongest performance came from a 34-billion-parameter model that made the login screen display the word “WORKING” in 96-point green text. This result was initially rejected by our evaluation harness because users could no longer log in. The model explained, through 14,000 tokens of commentary, that authentication is an implementation detail whereas visible reassurance is a product outcome.

This is difficult to argue with in a meeting, especially if the meeting has already exceeded 45 minutes.

Result two: “fix whatever is wrong” correctly identified that something was wrong

This prompt produced unusually high recall. Every model found at least one thing to fix, and several found things that had not yet become wrong. One replaced all nullable fields with non-nullable fields, eliminating an entire category of uncertainty by making the database reject reality. Another improved error handling by catching every exception and returning HTTP 200 with { "status": "vibes" }.

A smaller model was especially proactive. It noticed that the repository contained 1,846 TODO comments, including several written by engineers who had left the company, changed careers, or presumably become lighthouse keepers. It converted every TODO into a GitHub issue titled “Resolve Legacy Intent.” The resulting backlog was estimated to require 311 engineer-years, or one unusually focused afternoon after a quarterly planning meeting.

One model deleted a feature flag called enable_new_checkout and declared the checkout fixed. This was technically accurate in the sense that there was no longer a new checkout to malfunction. The model received a perfect score on our newly invented Metric for Reducing the Number of Things That Can Go Wrong by Removing Things.

Result three: “same thing but better” advanced the state of ambiguity

This was the most demanding prompt. “Same” requires preserving behavior. “Better” requires changing behavior. Together, they form the Schrödinger requirement: the system must be identical until observed by a stakeholder, at which point it must also feel more premium.

Models approached the task in different ways. One added gradients. One migrated the whole backend from Python to Rust, preserving an endpoint typo for compatibility. One rewrote a 27-line utility into 1,900 lines of generic abstractions, then added a comment reading // Improved version of previous implementation. A fourth made the button 4% rounder and categorized the release as a platform modernization.

The most operationally mature model created a directory called better/, copied the entire application into it, and updated no imports. This preserved the existing behavior completely while creating a clear future path. The future path was never taken, but its existence comforted everyone.

A practical guide to writing unusable coding prompts

Engineers wishing to reproduce these results should avoid accidental specificity. Do not mention the expected behavior, the actual behavior, a reproduction path, a stack trace, relevant files, constraints, acceptance criteria, or whether data loss is acceptable. Such details can bias the model toward solving a particular problem rather than demonstrating initiative.

• Replace “the payment form submits twice when the network retries” with “payments weird.”
• Replace “keep the public API stable” with “don’t break anything.”
• Replace “refactor this function for readability” with “clean this up, but not like last time.”
• If challenged, add “you have context” and close the laptop.

For maximum realism, answer every model clarification question with “use your judgment.” This lets the model exercise the judgment it acquired from billions of documents, including an unknown quantity of package READMEs written at 2:13 a.m. by people who had also been told to make it work.

The finding nobody can automate away

Our conclusion is that open-weight models are fully capable of turning vague requests into large, confident changes, which puts them in proud alignment with several established software processes. But a model cannot recover requirements that nobody has supplied. The true observation beneath the wreckage is simple: clear problem descriptions, constraints, and tests remain among the highest-leverage tools available to an engineering team.