If you are a lawyer, doctor, accountant, or anyone who handles sensitive documents, you have a problem. You want AI-powered analysis, but every cloud service copies your documents to their servers. Some use them for training. All of them are subject to breaches and subpoenas. There is a better way.

Why Cloud AI Is the Wrong Choice for Sensitive Docs

When you upload a contract to ChatGPT, that document lives on OpenAI's servers. Their privacy policy says they will not use it for training if you opt out, but it still traverses the internet, sits in their logs, and exists on infrastructure you do not control. For HIPAA-regulated data, attorney-client privileged communications, or financial records, that is a compliance nightmare.

The alternative is simple: run everything locally. Your documents never leave your machine. Period.

The Stack

You need three things:

  • A local LLM — Ollama running a model like Phi-3 Mini or Mistral 7B
  • Document extraction — Python libraries that pull text from PDFs and documents
  • A simple pipeline — Chunk text, generate embeddings, store in SQLite, query with the LLM

Step 1: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama pull phi3:mini
ollama pull nomic-embed-text

The first command installs Ollama. The second pulls a 3.8B parameter model for analysis. The third pulls an embedding model that converts text into vectors for search.

Step 2: Build the Document Pipeline

You do not need a vector database. For most personal document collections, SQLite with FTS5 full-text search is more than sufficient and infinitely simpler. Here is the core approach:

  1. Extract text from PDFs using PyMuPDF or pdfplumber
  2. Chunk the text into paragraphs (typically 200-500 words each)
  3. Generate embeddings using Ollama's nomic-embed-text model
  4. Store chunks and embeddings in SQLite
  5. At query time, find similar chunks via cosine similarity, then feed them to the LLM as context

The whole pipeline is under 200 lines of Python. No frameworks required.

Step 3: Process Your Documents

Point the script at a folder of PDFs. It will extract text, chunk it, embed it, and store everything locally. Processing a 50-page legal document takes about 10 seconds on modest hardware.

For a ready-made solution, PrivateGPT does most of this out of the box. You can also look at Cheeserag Studio or KnowledgeHub AI for more polished interfaces.

The Privacy Guarantee

When the entire stack runs on your machine, compliance becomes trivial:

  • HIPAA — no PHI leaves your control
  • GDPR — no data processing by third parties
  • Attorney-client privilege — documents never reach external servers
  • SOX, PCI-DSS — data stays in your infrastructure

This is not a theoretical benefit. This is the difference between "we use AI" and "we use AI and here is our compliance documentation."

Performance on Modest Hardware

The whole stack runs comfortably on 8GB of RAM with a 3B model. A typical query — "summarize the key obligations in this contract" — returns in 5-10 seconds. Not instant, but fast enough for real work.

If you have a GPU, inference drops to 1-3 seconds per query. The document processing step is mostly CPU-bound and finishes before you finish reading the document title.

When to Upgrade

The SQLite + FTS5 approach works well for collections under 10,000 documents. Beyond that, you might want a dedicated vector database like ChromaDB or Qdrant, both of which also run locally. But for individual professionals and small teams, the simple stack is the right starting point.

You do not need permission to use AI on your own documents. You just need to stop sending them to someone else's servers.

Related Posts