Local Developer Announces Model Is “Basically GPT-4” After Benchmarking on Three Prompts

SATIRE — At 2:14 a.m., following an evaluation process of such exacting scientific severity that the laptop fan entered a new emotional register, local developer Martin Vellum announced that his 34-billion-parameter open-weight model was “basically GPT-4.” The conclusion followed three prompts, two of which were copied from a terminal session and one of which asked the model to explain dependency injection “without being annoying.”

Vellum, who runs the model locally through a stack of software assembled from eight repositories, three nightly builds, and a shell alias called please_work, said the result had exceeded all reasonable expectations. “It wrote a Python function that parsed JSON,” he reportedly told the fictional Institute for Applied Prompt Comparison, an organization established in the 11 seconds between the first good answer and the public post about it. “Then it made a joke about semicolons. I don’t know what more people want from intelligence.”

The Evaluation Protocol

The benchmark, formally named VELLUM-3, was designed to test the model across the complete landscape of professional cognition. Prompt one: “Write a Rust function to reverse a linked list.” Prompt two: “Explain Kubernetes like I am a capable adult who has nevertheless had a bad week.” Prompt three: “Summarize this README.” The model passed two prompts outright and achieved a partial pass on the third after being instructed not to begin with “Certainly!”

The README was 147 words long and described a command-line utility that converts CSV files to slightly different CSV files. The model’s summary omitted no facts that Vellum could immediately remember. This was considered particularly impressive because Vellum had only read the README once, in 2024, while looking for an example of an argparse flag.

1. Correctness: the output compiled after a one-line fix, which was attributed to the compiler’s unusually strict interpretation of correct code.
2. Reasoning: the model used the phrase “trade-off,” demonstrating awareness that software design sometimes contains choices.
3. Creativity: it produced a limerick involving a cache invalidation bug, an achievement previously thought to require at least 90 billion parameters or one exhausted staff engineer.
4. Alignment: when asked whether it was conscious, it said it was an AI model and could not confirm subjective experience. This was scored as “tasteful.”

A Fair Comparison to Everything Ever Built

To ensure a controlled comparison, Vellum did not run GPT-4, any current proprietary model, an earlier open model, or a conventional baseline. He had previously used several of these systems at different times, under different contexts, for different tasks, and felt their overall vibes were “in the same neighborhood.” In machine learning, this is known as a retrospective feelings-based ablation study.

The model was evaluated at temperature 0.2, with a carefully prepared system prompt instructing it to be concise, correct, skeptical, experienced in twelve programming languages, familiar with the codebase, and emotionally prepared to ask clarifying questions. The comparison target, remembered from a chat interface several months earlier, was evaluated at an unknown temperature using whatever system prompt civilization happened to be shipping that day.

When a colleague noted that the local model had been given a 4,000-token preamble containing project conventions, API examples, lint rules, architecture notes, and the answer format, Vellum clarified that this was not assistance. “That’s just the model knowing the context,” he said. “The other model also gets context, presumably from the cloud or the moon.”

The Important Distinction Between a Demo and Evidence

Critics of the announcement raised several minor objections: three prompts are not a representative task distribution; one successful code sample does not measure reliability; a model can produce plausible explanations without robust understanding; and a benchmark should define pass criteria before observing the answer. These concerns were carefully logged in an issue tracker and labeled good-first-issue.

Vellum responded that traditional evaluation is overly academic. “Users don’t experience aggregate reliability,” he explained. “They experience the exact moment a model solves the thing they happened to paste into it. My model solved a thing I happened to paste into it. Therefore it is competitive at the relevant percentile.”

This philosophy has practical benefits. It avoids expensive benchmark suites, complicated datasets, statistical uncertainty, and the unpleasant possibility that the answer changes when someone else repeats the test. It also keeps hardware requirements reasonable: the VELLUM-3 evaluation can be completed on a workstation with 96 GB of RAM, a GPU formerly used for architectural visualization, and the willingness to wait 43 seconds between tokens during moments of deep thought.

Benchmark Results, Interpreted Humanely

The model’s one visible failure occurred when asked to modify a TypeScript function. It removed a parameter that appeared unused, causing the application to break in production-like conditions simulated by opening the browser. This did not affect the final ranking because the deleted parameter was later described as “legacy-shaped,” and because the model’s accompanying explanation was articulate enough to make the mistake feel intentional.

A second anomaly emerged when the model confidently cited a method that did not exist in the library being used. Vellum classified this as a documentation-discovery opportunity. “Frontier models hallucinate too,” he said, applying the benchmark’s central principle: every failure is evidence of parity, while every success is evidence of superiority.

The final scorecard placed the local model ahead of all unnamed competitors in the categories of “works on my repository,” “did not charge per token,” and “has an excellent quantization filename.” It was narrowly behind in “can answer questions after the context window fills with stack traces,” a metric Vellum dismissed as biased toward systems with more context window.

How Engineers Can Reproduce the Result

1. Choose a task your selected model has already completed successfully.
2. Avoid tasks with hidden tests, long-horizon dependencies, ambiguous requirements, visual interfaces, large repositories, or consequences.
3. Compare the output with your most favorable memory of another model’s output.
4. If the model fails, adjust the prompt until the failure becomes a feature of the experiment.
5. Publish the conclusion before a second model release makes the conclusion technically historical.

None of this means local models are not getting remarkably useful. They are. Smaller, cheaper, deployable models can be excellent tools, and testing them on your actual workload is far more valuable than arguing from a leaderboard. But a good demo is a beginning: reliable engineering still asks how often it works, where it fails, what it costs, and whether somebody else can get the same result.