Fine-Tuning on Your Own Codebase: What Teams Are Actually Reporting

Teams publicly reporting results from fine-tuning on their own code are finding the most value in narrow, testable jobs—not in turning a base model into a general expert on the whole repository. In a January report, Castform described training Qwen2.5-Coder-7B for one customer’s unit-test-generation workflow with a reward based on test coverage. The company says the resulting agent produced more realistic mocks and assertions and outperformed the general-purpose models it tested for that specific task.

What changed

The important distinction is between supplying repository context at inference time and changing model weights. Most coding-agent deployments still use the former: the agent searches the checkout, reads relevant files, follows repository instructions, runs tests, and returns a patch. That approach keeps the model general while letting it inspect the current branch, dependencies, and documentation.

Fine-tuning changes the model’s behavior using examples or reward signals. In the Castform case, the signal was not simply a dump of source files. It was whether a generated test increased coverage for a pull request or diff. That gives training a concrete target: fit the project’s testing conventions, fixture setup, mocks, assertions, and execution environment. It also creates a score that can be compared with a base model before deployment.

Microsoft Research’s Clean-PR work reaches a similar result in a broader setting. Its researchers converted validated pull-request histories into repository-editing training examples, then reported absolute gains of 13.6 percentage points on SWE-bench Lite and 12.3 points on SWE-bench Verified over an instruction-tuned baseline. The result does not demonstrate that an enterprise should train on its entire private repository. It does show why resolved changes, diffs, and validation outcomes are more useful training material than code snapshots alone.

Who this affects

This is relevant to teams with a repeated engineering task that has a reliable verifier: generating unit tests, migrating a framework pattern, producing a particular class of mechanical refactor, or proposing fixes whose acceptance can be checked by CI. It is less persuasive for ad hoc architecture work, product design, or broad codebase questions. Those tasks change often, need current context, and are poorly represented by a static training set.

What to do about it

1. Start with a baseline agent that can search the repository, use project instructions, and run the same checks a developer runs. Do not call a context problem a fine-tuning problem before testing retrieval and tooling.
2. Pick one task with an objective score: tests passing, coverage delta, static-analysis findings, migration completion, or accepted pull requests. Keep a held-out set of recent tasks for evaluation.
3. Train on successful change trajectories when possible: issue or diff, relevant context, patch, commands run, and test outcome. Remove secrets, generated artifacts, and examples that would not be acceptable for the model to reproduce.
4. Deploy the tuned model as a specialist behind the existing review and CI path. Compare cost, pass rate, regression rate, and human rework against the untuned agent before expanding its scope.