Running Local LLMs with Ollama

Install Ollama, pull a model, and call it from code through the same API shape as OpenAI's — all running entirely on your own machine.

Ollama packages an open-weight model, its runtime, and a simple API into one command — ollama run llama3.2 and you're talking to a model with no account, no API key, and no data leaving your machine. This is the fastest path to experimenting with LLMs locally, no transformers/torch wiring required.

Install Ollama

macOS:

brew install ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: download the installer from ollama.com/download.

Start the background service (macOS/Windows apps do this automatically; on Linux it's usually installed as a systemd service already running):

ollama serve

Pull and run a model

ollama pull llama3.2
ollama run llama3.2

run drops you into an interactive chat right in the terminal. Type a message, get a response, /bye to exit.

tip

Model names include a size when there's a choice — llama3.2:1b vs. llama3.2:3b, for instance. Smaller = faster and less RAM, larger = better quality. ollama pull <model> with no tag pulls a sensible default.

Picking a model size for your hardware

Model size	Rough RAM needed	Realistic on
~1–3B params	4–8GB	Any recent laptop, CPU-only
~7–8B params	8–16GB	Laptops with 16GB+ RAM, or any GPU
~13B+ params	16–32GB+	Dedicated GPU strongly recommended

These are quantized (compressed) sizes — Ollama pulls a quantized version by default, which is why a "7B" model doesn't need anywhere near the memory naive parameter-count math would suggest.

ollama list        # models you've pulled
ollama ps           # what's currently loaded in memory
ollama rm llama3.2  # free up disk space

Calling it from code

Ollama exposes a local HTTP API on port 11434 — and, usefully, an OpenAI-compatible endpoint, so existing OpenAI-client code often works with just a base URL change:

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by the client, unused by Ollama
)
 
response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain what Ollama is in one sentence."}],
)
print(response.choices[0].message.content)

Or hit the native API directly:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain what Ollama is in one sentence.",
  "stream": false
}'

Common gotchas

warning

Generation is extremely slow. You're almost certainly running a model too large for CPU-only inference. Check ollama ps — if it shows 100% CPU with no GPU listed, drop to a smaller model size or confirm your GPU drivers are actually being picked up (nvidia-smi on Linux/ Windows should show activity while a model runs).

warning

"connection refused" calling the API. The Ollama service isn't running. ollama serve in a terminal you leave open, or confirm the background app/service is actually active.

note

Ollama is for running models, not training or fine-tuning them. For experimenting with prompts and API shape, it's the fastest path; for fine-tuning workflows you're back to the full LLM development environment setup with transformers and a real GPU.

With a model running locally and an OpenAI-compatible endpoint in front of it, you can prototype a feature entirely offline, then swap in a hosted API later by changing a base URL and a key — not by rewriting the integration.