Everyone thinks you need a $3,000 GPU rig to run local AI. That was barely true in 2024 and it is definitely not true in 2026. Quantized models have gotten shockingly good, and a new class of Mixture-of-Experts models has changed the math entirely. I have been running LLMs on old hardware for over a year now, and here is what actually works.

The Hardware You Already Have Is Enough

The key insight is quantization. A quantized model at Q4_K_M precision uses roughly 0.6GB of RAM per billion parameters. That means a 7B parameter model at Q4 needs about 4.5GB of RAM total. If you have an 8GB laptop from 2019, you can run it with room to spare for the OS.

On a 4GB machine, you are looking at models up to 3B parameters comfortably. On 8GB, you can push to 7B-8B models. Both are genuinely useful for code completion, summarization, Q&A, and drafting.

The Models Worth Running

Here is what I recommend based on actual RAM availability:

  • Phi-3 Mini (3.8B, 2.3GB download) — Microsoft's small model punches well above its weight. Excellent for code and reasoning tasks.
  • Gemma 2 2B (1.5GB download) — Google's compact model is fast and surprisingly capable for its size.
  • TinyLlama 1.1B (700MB download) — The fastest option on tight hardware. Great for quick completions and simple tasks.
  • Qwen 2.5 3B (2GB download) — Strong multilingual support and solid coding ability.

MoE Models Changed Everything

Mixture-of-Experts models are the real game-changer in 2026. A model like Qwen 3.5 35B-A3B has 35 billion total parameters but only activates about 3 billion per inference step. The practical result: you get near-35B quality at 3B computational cost.

These models fit in 6GB of VRAM or even 8GB of system RAM with some swap. The quality jump over small dense models is dramatic. If your hardware supports it, always prefer a MoE model over a dense model of similar active-parameter count.

Performance Numbers

Real-world tokens-per-second on constrained hardware:

Model4GB RAM8GB RAMGPU (6GB VRAM)
TinyLlama 1.1B8-12 t/s12-18 t/s40+ t/s
Gemma 2 2B5-8 t/s10-15 t/s35+ t/s
Phi-3 Mini 3.8B3-5 t/s8-12 t/s25+ t/s
Qwen 2.5 3B4-6 t/s8-12 t/s30+ t/s

For an interactive chat experience, anything above 5 tokens per second feels responsive. You will get that on 8GB with most of these models.

The Setup

Ollama is the easiest path. Install it and you are running models in under a minute:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull phi3:mini
ollama run phi3:mini

For power users, llama.cpp gives you more control over quantization, context length, and memory management. The --n-cpu-moe flag lets you split MoE expert layers between CPU and GPU, which is critical on mixed hardware.

Optimization Tricks

Three changes that make the biggest difference on tight hardware:

  • Cap context to 4K tokens. KV cache memory scales linearly with context length. Dropping from 8K to 4K cuts memory usage significantly.
  • Use KV cache quantization (q8_0). This cuts cache memory roughly in half with minimal quality loss.
  • Add swap on Linux. A 4GB swap file on an SSD gives you breathing room without crashing when you push model size.

On Linux, also close your browser before running inference. Chrome alone can eat 2-3GB of RAM.

The Honest Take

Can you run a 70B parameter model on your old laptop? No. But you can run a 3B MoE model that handles most real-world tasks with surprising competence. The gap between "runs locally" and "runs GPT-4 level" is closing every quarter. For writing, coding, and analysis tasks, what you can run on a 2019 laptop today is genuinely useful — not just a toy demo.

Related Posts