How to Run AI Coding Agents Locally Without Sending Your Code to the Cloud

Yes: run the model on your own machine, connect your coding agent to a localhost endpoint, and it can inspect and edit code without uploading your repository to a model API. But don’t mistake that for automatic privacy: a local model prevents model-prompt traffic from leaving your computer, while the agent’s tools, editor extension, telemetry, and shell commands may still have their own network paths.

The practical setup is simpler than it sounds. You need a local inference server, a model that can reliably call tools, and an agent interface that can use that server instead of an API key. Ollama plus Continue is a useful first stack because Ollama exposes a local endpoint and Continue can use local Ollama models for chat, editing, and agent workflows.

Understand the three pieces before installing anything

A coding agent is not just an LLM in a sidebar. The model decides what to do; the agent harness reads files, applies patches, runs commands, and feeds results back to the model; the inference server turns prompts into model output. For an actually local workflow, all three need to run on hardware you control.

Ollama is the inference layer in this example. It downloads and serves a model locally, usually on http://localhost:11434. Continue is the agent interface in your editor. It sends selected code, file contents, diffs, terminal output, and your instructions to the configured model endpoint. If that endpoint is localhost, those prompts stay on the machine rather than going to a hosted LLM provider.

There is one unavoidable exception: you must download the software and model weights at least once. That is not the same as sending your source code away. Once the model is installed, you can disconnect the machine from the internet and still run inference locally.

Build a minimal local agent stack

Start with Ollama installed on the workstation that holds your repository. Then pull a coding-capable model with tool-use support. Tool use matters because agent mode needs the model to request actions such as reading a file, searching the codebase, or running a test; a good chat model without reliable function calling will often produce advice rather than complete an agent loop.

# Download a local coding model
ollama pull qwen3-coder

# Confirm the local service answers on loopback
curl http://localhost:11434

# See downloaded models
ollama list

Next, install Continue in VS Code and add an Ollama-backed model to its local configuration. The exact configuration surface can vary by Continue release, but the critical values are the provider, the exact model name shown by ollama list, and a localhost API base. If your chosen model supports function calling, explicitly enabling tool use can help when capability detection gets it wrong.

name: Local Coding Agent
version: 0.0.1
schema: v1

models:
  - name: Local Qwen Coder
    provider: ollama
    model: qwen3-coder
    apiBase: http://localhost:11434
    capabilities:
      - tool_use

Open a small, disposable repository first. In Continue, choose the local model, switch to agent mode, and ask for a bounded task: “Find the failing unit test, explain the likely cause, then propose a patch. Do not execute commands until I approve them.” This verifies the whole loop—context collection, tool calls, edits, and test feedback—without handing an autonomous process your production repository on day one.

Pick the model for the job your machine can sustain

Local coding quality is constrained by memory, speed, context length, and tool-call reliability. Bigger is not automatically better if each response takes long enough that you stop reviewing the agent’s work. A smaller model can be excellent for narrow edits, test failures, and code explanation; larger coding models generally make more sense for multi-file refactors and longer debugging sessions.

Use a short context window while bringing the system up. Large contexts consume memory and make every agent turn more expensive. An agent does not need to ingest your entire monorepo to rename an API field. Give it a target directory, the relevant failing test, and a crisp acceptance condition. This also improves privacy because the agent has less source material to process at all.

If Ollama reports that the model requires more system memory, do not immediately chase exotic runtime flags. First reduce the context length, choose a smaller variant, close GPU-heavy apps, and retry the same small task. Local agents are useful when they are dependable, not when they barely fit.

Close the holes that a local model does not close

A local endpoint is necessary, but it is not a complete no-egress policy. Treat the agent like an untrusted developer account with terminal access. It may run npm install, call a remote MCP server, use a web-search tool, fetch documentation, invoke a cloud CLI, or include secrets in terminal output that the model can then read.

• Bind the model server to loopback only. Do not expose port 11434 to your LAN or the public internet unless you add authentication and network controls.
• Disable web search, hosted MCP servers, and cloud-provider tools for repositories that must remain local. A remote tool can receive prompts or file-derived context even when the model is local.
• Require approval for shell commands and file writes at first. Allow read-only exploration freely, then approve edits and test commands deliberately.
• Use a repository-specific .gitignore and agent rules that prohibit reading .env, credential stores, production exports, and private keys.
• Inspect extension settings for telemetry and crash reporting. If your policy requires no outbound traffic, enforce it with a host firewall or an isolated VM rather than trusting a checkbox.
• Run the agent in a container or VM for higher-risk tasks, with the repository mounted read-only until you are ready to accept a patch.

Use a workflow that makes local agents worth the trade-off

The strongest local-agent workflow is review-first, not autonomous-first. Ask the agent to inspect, summarize, and produce a patch plan. Then let it edit a narrow set of files. Finally, have it run a targeted test command and show the diff. You still get fast codebase navigation and repetitive implementation help, while keeping the point of control where it belongs: with the engineer who knows what the system is supposed to do.

You should also test your privacy claim rather than assume it. Start a packet monitor or temporarily block outbound traffic, then run a normal agent task. The model should continue working after its weights are installed. If it fails only when the network is blocked, identify whether the cause is telemetry, an extension service, a remote tool, dependency resolution, or a model configuration that quietly points at a hosted endpoint.

That last test is the difference between “I selected a local model” and “this repository stays on my machine.” The first is a configuration choice. The second is an engineering property you can observe, constrain, and verify.