LLM Development Environment Setup

Python, the libraries you actually need, GPU vs. CPU decisions, and — most importantly — keeping API keys out of your Git history.

Building with LLMs — whether calling a hosted API or running models locally — is mostly a normal Python setup with a couple of LLM-specific decisions layered on: which libraries you need, whether you have a GPU worth using, and how you keep provider API keys from ending up in Git history. This guide covers all three.

This assumes the Python environment setup guide — pyenv and virtual environments — is already in place.

Set up an isolated environment

Every LLM project should get its own virtual environment — the dependency tree here (torch, transformers, and friends) is large and version- sensitive enough that you don't want it anywhere near an unrelated project's environment.

pyenv local 3.12
python -m venv .venv
source .venv/bin/activate

Calling hosted APIs (OpenAI, Anthropic, etc.)

If you're calling a hosted model rather than running one locally, this is most of what you need:

pip install anthropic openai python-dotenv

Keep API keys in a .env file, never in code:

# .env
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

echo ".env" >> .gitignore

from dotenv import load_dotenv
import os
 
load_dotenv()
api_key = os.environ["ANTHROPIC_API_KEY"]

warning

A leaked API key is a "rotate it now" problem, not a "fix it later" one. If a key ever gets committed — even briefly, even in a private repo — treat it as compromised: revoke it in the provider's dashboard and issue a new one. git history keeps it forever even if you delete the file in a later commit.

Running models locally: the extra decisions

If you're running open-weight models yourself (rather than calling a hosted API), you need torch and, for most workflows, transformers:

pip install torch transformers accelerate

Check whether you actually have GPU acceleration

import torch
print(torch.cuda.is_available())       # NVIDIA GPUs (Linux/Windows)
print(torch.backends.mps.is_available())  # Apple Silicon

warning

torch.cuda.is_available() returns False on a machine with an NVIDIA GPU. The most common cause is installing the default pip install torch build, which on some platforms resolves to a CPU-only wheel. Install from the exact command on pytorch.org's get-started page — it generates the right pip install invocation for your OS and CUDA version, which is not always the plain package name.

Without a capable GPU, small models (a few billion parameters, quantized) run tolerably on CPU or Apple Silicon; anything larger becomes painfully slow. If that's your situation, running models through Ollama — which handles quantization and hardware detection for you — is a much smoother path than wiring up transformers directly.

An environment for experimenting: Jupyter

Notebooks are the natural format for iterating on prompts and inspecting model output interactively:

pip install jupyterlab
jupyter lab

Pin what you install

As the environment grows, freeze it so it's reproducible:

pip freeze > requirements.txt

tip

For anything beyond a quick experiment, consider uv or Poetry instead of raw pip freeze — LLM-adjacent dependency trees resolve slowly with plain pip, and a lockfile-based tool avoids the "works on my machine" version drift this space is especially prone to.

Common gotchas

warning

pip install torch takes forever or fails on disk space. GPU-enabled PyTorch wheels are large (several GB) — a slow install isn't necessarily broken, but do check available disk space first if it fails partway.

warning

Out-of-memory errors loading a model. Model size in parameters roughly maps to memory needed — a 7B parameter model needs roughly 14GB in full precision, much less (often a quarter) in 4-bit quantization. If you're hitting OOM, that's the first thing to check: are you loading a model too large for the box, at a precision higher than you need?

With keys safely in .env, a virtual environment isolating the dependency tree, and a clear-eyed read on your actual hardware, you're set up to experiment with either hosted APIs or local models without the usual first-week setup friction.