Startup Releases Weights, Training Code, and Absolutely No Way to Reproduce Any of It

SATIRE — Today, the startup Verifable-ish Intelligence released the weights and training code for its new 48-billion-parameter model, GlassMantis-48B, proving once and for all that reproducibility is not a destination, a process, or even a property of an experiment. It is a tasteful word to place near a GitHub badge. The release gives engineers everything they need to reproduce the original run except the data, data-processing pipeline, tokenizer version, training configuration, scheduler settings, random seeds, infrastructure setup, checkpoint cadence, failed-run history, and the particular lunar phase under which the loss curve became emotionally available.

A Complete Release, Minus the Parts That Matter

The model card explains that GlassMantis-48B was trained on “a diverse, carefully curated corpus of high-quality text, code, documents, conversations, structured information, semi-structured information, and several forms of information not yet classified by science.” The corpus itself cannot be shared, both because of licensing constraints and because it was assembled from 19,000 temporary cloud disks that automatically deleted themselves after achieving inner peace.

To ensure scientific rigor, Verifable-ish has released a file called train.py. It contains 312 lines of import statements, a function named run_training(), and a TODO reading “insert proprietary dataset interface here.” The function then invokes an internal package called nebula_furnace, which is not included in the repository but is described in the README as “broadly similar to PyTorch, except operational.”

The repository includes a requirements.txt with one dependency: torch>=2.0. It also includes a Dockerfile that begins FROM internal-registry/please-do-not-share:latest. Engineers attempting to build the image are told that this is expected behavior and demonstrates the project’s commitment to protecting the integrity of the environment.

The Dataset Is Open in Spirit

Asked whether researchers could inspect the data mixture, the company published a pie chart with five slices: Web, Code, Books, Multilingual, and Other. The Other slice occupies 63 percent of the chart and is shaded with a gradient that cannot be sampled by ordinary image editors. A footnote clarifies that “Web” should not be interpreted to mean the web, “Books” should not be interpreted to mean books, and “Code” includes any sequence of characters that caused an engineer to nod thoughtfully.

The data-cleaning procedure is equally accessible. According to the technical report, the team removed low-quality content using a multi-stage quality system. Stage one was a classifier. Stage two was another classifier. Stage three was “additional measures.” The report does not describe the classifiers, the thresholds, the language coverage, the deduplication method, or whether the same document was allowed to appear 400 times if it had a sufficiently entrepreneurial tone.

This omission is not a problem, the company says, because no serious researcher would want to recreate a training dataset exactly. Exact recreation could lead to inconvenient questions such as why a model trained on a public corpus knows the first 11 paragraphs of a paywalled plumbing forum from 2017, including the moderator’s edit about galvanized fittings.

Training Code Has Been Provided for Educational Purposes

The released training code demonstrates the general shape of large-scale training. First, it initializes distributed execution across a variable named WORLD_SIZE, which must equal the number of accelerators available in the original experiment. This number is not disclosed, but the documentation says it was “more than one, fewer than astronomy.” Next, it loads a configuration object from a path supplied by the deployment system. Finally, it begins training after confirming the presence of an environment variable called THE_REAL_CONFIG.

There are no hyperparameters in the repository. Instead, there is a JSON file containing the values null, “redacted,” “see launch settings,” and 0.0000000000000000000000000000000000000000000000000001. This last value is believed to be either the learning rate or the probability that an outside lab can match the reported benchmark score.

The team has thoughtfully published one random seed: 42. The remaining seeds are stored in a hardware security module that was melted down after the final checkpoint, in accordance with the company’s Data Minimalism Initiative. This is not secrecy. It is a reproducibility enhancement, because nobody can accidentally reproduce the run incorrectly.

Independent Verification Is Encouraged Within Approved Limits

Verifable-ish welcomes outside replication efforts. To help, it has provided the final weights, which are sufficient to run inference and insufficient to answer nearly every question about how the model came to behave as it does. Researchers may compare outputs, evaluate downstream performance, inspect architecture choices, and speculate professionally about the rest.

Anyone who achieves materially different results is encouraged to check whether they used the exact undocumented tokenizer revision, the exact unavailable data order, the exact missing optimizer state, and the exact cluster topology, including a single network card that reportedly delivered packets with “just the right amount of uncertainty.” Anyone who achieves similar results is invited to celebrate the robustness of the method, even if their model was trained on a different corpus, a different codebase, and a laptop during a delayed train.

What “Open” Can Still Mean

To be fair, publishing weights is useful. It lets engineers run, inspect, fine-tune, benchmark, and deploy a model without asking a vendor for permission at every token boundary. Publishing training code can be useful too, even when it is incomplete: it can reveal architectural decisions and invite scrutiny of at least some implementation choices.

But reproducibility is a stricter claim. Training results depend on more than a final checkpoint and a file called train.py: data provenance and preprocessing, configurations, software versions, evaluation details, compute setup, and enough experimental records to explain what actually happened all matter. The joke is that these boring artifacts are often the most valuable part of an “open” release.