Run your first local AI in 30 minutes on Windows
You can have a private AI chatting with you on your own Windows PC in about 30 minutes, using a free tool called Ollama — no cloud account, no subscription, no data leaving your machine. This walks through the install, picking a model your hardware can actually run, your first chat, and the one setting you should never change without knowing why.
TL;DR
- Install Ollama from ollama.com/download/windows — a small installer, no admin rights needed, Windows 10 or later. ⏱️ verified 2026-07-05
- Pick a model sized to your GPU's VRAM, not the biggest name you've heard of. See VRAM: the one number that decides which AI models you can run for the sizing logic.
- First chat takes one command:
ollama run <model>downloads the model (first time only) and drops you into a chat prompt — or use the desktop app's chat window if you'd rather not touch a terminal. - Expect real limits, honestly: a small model on a modest GPU is fast but not brilliant; a good model on no GPU at all is usable but slow. Match your expectations to your hardware, not to demos of much bigger machines.
- Do not expose the server to your network without understanding what that means — Ollama has zero built-in authentication. Covered in the warning box below and in why running AI locally is a security decision.
What am I actually installing?
Ollama is a free program that downloads AI models to your PC and runs them using your CPU and (if you have one) your GPU. It handles the technical parts — model formats, memory management, an interactive chat — behind one simple command, which is why it's the default starting point for local AI on Windows.
It's not the only way to run models locally, and it's not the fastest for every job, but it's the lowest-friction way to get from "nothing installed" to "having a private conversation with a model" — which is the point of this walkthrough.
How do I install it?
Go to ollama.com/download/windows and download OllamaSetup.exe. Requirements are modest: Windows 10 or later, and if you have an NVIDIA GPU, driver version 551.61 or newer (AMD needs ROCm 7 or a Vulkan-capable driver) for GPU acceleration — Ollama still runs on CPU-only machines, just slower (Ollama Windows docs, ⏱️ verified 2026-07-05).
- Run
OllamaSetup.exe. It installs to your user account — no administrator rights required, no UAC prompt. - Let it finish. Ollama starts automatically in the background and adds an icon to your system tray (bottom-right, near the clock).
- Allow the firewall prompt if you see one, choosing "private networks." This just lets local apps on your PC (like a chat interface) talk to Ollama on your machine — it does not open anything to the internet.
- Open a new terminal window (PowerShell or Command Prompt) if you plan to use the command line — a terminal that was already open before you installed won't see the updated PATH.
That's it. No account, no license key, no reboot required in the normal case.
How do I have my first chat?
You have two options, and both are legitimate — pick whichever fits how you like to work.
Option A — the desktop app. Click the Ollama tray icon or its Start-menu shortcut to open the built-in chat window, type a message, and go. This is the easier on-ramp if you'd rather not touch a terminal.
Option B — the terminal, which doubles as your first lesson in how Ollama actually works:
ollama run llama3.2
The first time you run this, it downloads the model (a few gigabytes, so give it a few minutes on a normal connection); every time after, it starts instantly. Once it's loaded, you're in an interactive chat — type a message, press Enter, get a response, keep going. Type /bye to exit.
llama3.2 here refers to a small, capable general-purpose model (Meta's Llama 3.2 3B) that runs comfortably on almost any GPU with a few gigabytes of VRAM, or even CPU-only — a sensible default for your very first run. The next section covers picking something bigger if your hardware supports it.
Which model should I actually run?
The honest answer: whatever fits your GPU's VRAM, because a model that doesn't fit doesn't run slightly worse — it runs dramatically worse. This is the single most important decision in this whole process, and it's covered in full in VRAM: the one number that decides which AI models you can run. The short version, at the standard 4-bit quantization current Ollama models ship with by default:
| Your GPU VRAM | Model class that fits well | Example |
|---|---|---|
| No dedicated GPU / shared memory | 1–3B | llama3.2, gemma3:1b |
| 8 GB | 7–8B | llama3.1:8b, mistral |
| 16 GB | up to 14B | qwen3:14b, phi4 |
| 24 GB | up to 32B | qwen3:32b, deepseek-r1:32b |
Pull any of these the same way — ollama run <name> — and Ollama downloads and runs it. If you're not sure how much VRAM your card has, open Task Manager → Performance → GPU, or run nvidia-smi in a terminal on an NVIDIA card.
⚠️ Don't expose your Ollama server to the network
Ollama binds to 127.0.0.1 (your own machine only) by default, on port 11434 — nothing outside your PC can reach it unless you change that (Ollama FAQ, ⏱️ verified 2026-07-05). Some guides show you how to set the OLLAMA_HOST environment variable to 0.0.0.0 so other devices on your network — or the internet — can reach it, usually to let a phone or a second PC use the same server.
Do not do this without a plan, because Ollama ships with no built-in authentication at all. Anyone who can reach the address can run prompts, pull and delete models, and consume your GPU — there's no login screen to stop them. Researchers found roughly 175,000 Ollama servers sitting open on the public internet with zero auth, in active use by attackers, because people changed this setting without realizing what they'd removed. The full story, including what those attackers actually did with it, is in why running AI locally is a security decision.
If you do want other devices on your home network to reach your server later, that's a reasonable goal — just do it behind a firewall rule or a reverse proxy with authentication in front, never a bare 0.0.0.0 bind facing your router. For your first 30 minutes, leave the default alone.
What speed and quality should I honestly expect?
It depends entirely on your GPU, and the rule from the VRAM post applies directly: speed roughly tracks memory bandwidth ÷ model size. A small model (3B, roughly 2 GB at Q4) on almost any GPU from the last several years feels instant — comfortably faster than you can read. An 8B model on a mid-range 8–16 GB card is still fast, well above reading speed. A 32B model on a 24 GB card is usable but noticeably more deliberate — you'll see words arrive, not appear instantly.
CPU-only, no GPU: it works, but budget for slow. System RAM bandwidth is a fraction of a GPU's, so even a small model can drop to a few words per second — fine for testing the idea, tedious for daily use. If that's your situation, start with the smallest model (llama3.2:1b or gemma3:1b) so you're judging the experience, not fighting your hardware.
On quality: a 3–8B model handles everyday chat, summarizing, and simple writing help well, and will occasionally be confidently wrong on facts or math — verify anything that matters. It is not a substitute for a large frontier cloud model on hard reasoning or complex coding tasks; it's a private, free, always-available assistant for the tasks that don't need that ceiling.
When this won't help
- You want frontier-model quality on modest hardware. Local models at sizes that fit consumer GPUs trail the largest cloud models on hard reasoning, complex coding, and up-to-date world knowledge. Local wins on privacy, cost, and availability — not on raw capability at every task.
- You need a shared, multi-user server today. This guide gets one person chatting on one PC. Serving your whole household or exposing the API safely is a separate, deliberate networking project — don't wing it with a raw port opened to your LAN or the internet.
- Your GPU has under ~4 GB of VRAM (or none) and you need serious speed. It'll run, but slowly enough that CPU-only users especially should treat this as a proof-of-concept, not a daily driver, until better hardware is in the picture.
- You want a nicer chat interface with history, multiple models, and file uploads. The built-in desktop app and terminal are enough to start, but a dedicated web UI (a separate setup) is the next natural step once you're comfortable.
FAQ
Do I need a graphics card to run Ollama? No — it runs on CPU-only Windows PCs. It'll work for small models, but expect noticeably slower responses than on a machine with a dedicated GPU, since system RAM bandwidth is much lower than GPU VRAM bandwidth.
Is Ollama free? Yes. Ollama itself and the models distributed through it (Llama, Qwen, Gemma, Mistral, and others) are free to download and run locally.
How do I know how much VRAM my GPU has?
On Windows, open Task Manager → Performance tab → select your GPU → look for "Dedicated GPU memory." NVIDIA users can also run nvidia-smi in a terminal.
Can other people on my home network use my Ollama server?
Only if you deliberately change the default binding (OLLAMA_HOST) — by default it's locked to your own machine. Don't do this until you've read the security warning above and put a firewall rule or authenticated proxy in place.
What's the difference between the desktop app and the terminal?
Same underlying engine — the desktop app gives you a graphical chat window, the terminal (ollama run <model>) gives you the same chat in text form plus access to every other command (pulling models, listing what's installed, checking status). Neither is more "correct"; use whichever you're comfortable with.
Last updated 2026-07-05. Time-sensitive: Ollama's version, default model recommendations, and the exact library of available models change frequently — check ollama.com/library for what's current before you pick a model.
Sources (verified 2026-07-05): install steps and requirements — Ollama Windows download, Ollama Windows docs; default network binding and OLLAMA_HOST — Ollama FAQ; run command behavior — Ollama quickstart; exposed-server research — The Hacker News.