You can run a language model on your own computer without sending every question to a remote AI service. A small model, a compatible application and some free memory are enough to start experimenting. No server room required; the laptop fan may still have an opinion.
The useful first goal is modest: get one local model working, try it on a task you understand and see whether the result is worth the resources it uses. This guide takes that route, with established tools and an explicit model name rather than a shopping list of expensive hardware.
What actually runs locally?
A language model contains numerical weights learned during training. Inference is the computation that uses those weights and your input to generate an answer, usually a token at a time. Tokens can represent parts of words as well as whole words or punctuation.
With local inference, that computation happens on your device. The application may use the CPU, a supported GPU or a combination. It is separate from training a model from scratch, which is a much larger undertaking.
Check the model and application settings before assuming the whole workflow is local. Ollama introduced cloud-hosted models in September 2025, for example. The same familiar interface can therefore use either your machine or remote computing resources. A cloud model does not become local because you launched it from a terminal.
Memory: leave room beyond the model file
Quantisation reduces the precision used to represent model values, lowering memory requirements. It can also affect output quality and performance; the trade-off depends on the method and model.
For a rough illustration, seven billion weights stored at 16 bits require about 14 GB just for those weights. At four bits, the arithmetic gives about 3.5 GB before additional data and overhead. Neither figure is a complete system-memory requirement.
The runtime needs working space, and the conversation’s context consumes memory too. Your operating system and other applications also need their share. A model download fitting on disk does not establish that a long conversation will fit comfortably in RAM or VRAM.
A practical approach is to start with a small quantised model and a short context. Check the application’s hardware requirements and memory estimate. If the machine begins swapping heavily or responding painfully slowly, step down in size before considering a purchase.
There is no universal 70B ceiling for home use, nor a guarantee that any particular parameter count will run well on every machine with the same amount of RAM. Memory bandwidth, acceleration support and the model’s architecture all matter.
Choose an interface you will actually use
Ollama: a straightforward terminal route
Ollama manages model downloads and provides a way to run them from the command line or a local API. Its Llama 3.2 library page gives a compact starting example, used below. Choose a downloadable local model for this exercise.
LM Studio: a graphical route
LM Studio offers a model browser and chat interface. Its offline-operation documentation says downloaded models, local document processing and a local inference server can work without internet access. Model searches, downloads and update checks require connectivity.
If a graphical interface feels more comfortable, use that. Learning a command line is optional; understanding where your model runs is not.
llama.cpp: more direct control
llama.cpp is a C/C++ inference project with quantised-model support, CPU execution and several hardware-acceleration options. It also supports splitting work between CPU and GPU. It is useful for people who want closer control over configuration.
No one of these tools is automatically fastest on every computer. Start with the simplest option that supports your hardware and intended model.
Your first local chat with Ollama
- Install the application. Use the installer or instructions for your operating system from Ollama’s official download page. Allow time for the installation and model download; connection speeds vary.
- Download an explicit small model. Open a terminal and enter
ollama pull llama3.2:3b. Ollama lists this model’s default quantised download at roughly 2 GB. That is the download size, not the total memory it will use. - Start it. Enter
ollama run llama3.2:3b. For a smaller starting point, the library also providesllama3.2:1b. - Try a simple task. Ask for a short rewrite of a paragraph you wrote, then compare the result with the original. Check whether it preserved names, numbers and meaning.
- Exit the chat. Type
/bye. You can return to the same model later without downloading it again.
Llama 3.2’s text models are 1B and 3B, not 8B. Meta’s model card describes their intended uses and licence. Here, the 3B version is a manageable example rather than a claim that a 2024 release is the best model for every task in 2026.
Once installation and downloads are complete, a basic chat with this local model can work offline. If you attach tools that need the internet, those tools will have their own requirements.
Try different tasks before trying bigger models
A model that produces a charming paragraph may still make arithmetic errors or invent a software function. Evaluate it on examples whose answers you can check. For coding, use a small test project and inspect the changes before running generated code.
An established coding-focused alternative is Qwen2.5-Coder-7B-Instruct. Check that your runner supports the particular format and quantisation you download. A model’s name, parameter count, file format and licence answer different questions.
The wider competition between AI coding tools is a reminder that a useful workflow involves more than a benchmark score. Choose for the tasks you actually do, not the most impressive number you can persuade the computer to load.
Fresh information and privacy depend on the workflow
A model’s weights do not automatically acquire yesterday’s news. You can supply documents or connect retrieval and search tools. Ollama’s September 2025 web-search announcement is one example of that extra layer.
Those connections change the privacy picture. A remote search request or cloud integration can send information outside your machine even while the language model itself runs locally. Local chat files may also be included in your computer’s backups or accessible to other software.
For a basic private setup, use a local model, understand which integrations are enabled and keep the machine itself secure. Confidential material deserves a review of the entire workflow, not just the word “local” on a product page.
Costs and the local API
You can start with freely available tools and model weights, subject to their licences. Running them still uses hardware, electricity, storage and your time. Whether that costs less than a hosted service depends on usage; it is not guaranteed to pay for a new computer.
Developers can connect applications to Ollama’s local server. The Ollama FAQ documents the default address, 127.0.0.1:11434. Keeping it on the local machine is sufficient for this introductory setup. Making it reachable from other devices introduces a separate access-control task.
The reward is control and a useful place to experiment. Start small, keep the answers under review and expand only when a real task calls for it. The model does not need to be enormous to help draft an email. The cat, naturally, will still insist on editing with all four paws.
🐾 Curiosity looks good on you. Explore the Goodies, or find our illustrated books on Amazon.
