Self-hosted AI: a local model or an API key?

Chat apps such as LobeChat and AnythingLLM are only the front end. This guide explains where the model runs, what each choice costs, and which one a beginner should pick first.

The app is not the model

LobeChat and AnythingLLM give you a chat page, saved conversations and, in AnythingLLM's case, document search. They do not contain a large language model (LLM, the program that writes the answers). They send your question to one. You have two places to send it:

1. Your own hardware. A program such as Ollama runs an open model on your machine. Nothing leaves your network.
2. A paid provider. You make an account at OpenAI, Anthropic, OpenRouter or similar, and you get an API key (a long secret string). The app sends each question to the provider. The provider bills you per token. A token is a small piece of a word, and you pay for the ones you send and the ones you get back.

Both apps accept either. AnythingLLM lists local providers (Ollama among them) and cloud providers, and lets you connect to both at the same time. LobeChat, now named LobeHub, has a README that walks through an OpenAI key first, and a setup page for Ollama.

Option 1: run a model on your own hardware

Memory dictates which models you can run. A model is a large file of numbers (weights). The whole file must sit in memory while it answers. On a graphics card (GPU), that memory is called VRAM. Without a GPU, the model uses normal RAM.

Rule of thumb. The model's file size is the floor, and you need extra room for the conversation itself. Older Ollama READMEs gave a simple guide: at least 8 GB of RAM for 7B models, 16 GB for 13B models and 32 GB for 33B models. The "B" means billions of parameters. The current README no longer repeats this, so treat it as a rough rule.

Quantization changes the size. It stores each number with fewer bits, which shrinks the file and costs a little quality. The Llama 3.1 tags on Ollama's library show the effect, as of October 2026:

A longer conversation needs more memory. Ollama says a larger context length raises the memory a model needs and defaults to a 4k context below 24 GiB of VRAM to keep this in check.

What this means in practice:

• A CPU-only machine works, but slowly. Ollama can load a model entirely into system memory, and replies then arrive word by word at a crawl. Try it for a small model, but do not build a daily habit on it.
• A Raspberry Pi or a small VPS is not a fit. Most have a few GB of RAM and no GPU. The model alone would fill the memory, and the app would have nothing left.
• Each model takes several GB of disk. Pulling three or four models fills a small disk. See where your data lives for how to plan space.
• A GPU makes it usable. Ollama supports NVIDIA cards with compute capability 5.0 or higher and driver 550 or newer, plus AMD and Apple GPUs.

GPU passthrough in Docker

A container cannot see the GPU by default. For NVIDIA cards you install the host driver, then the NVIDIA Container Toolkit, then tell Docker to use it. Ollama's own Docker page gives the steps for Debian and Ubuntu. After the toolkit package is installed, run:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then reserve the GPU for the Ollama service in Compose. The tag pins a version, as Update without breaking advises. Check the current one. The Docker docs require the capabilities field:

services:
  ollama:
    image: ollama/ollama:0.40.11
    volumes:
      - ./ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

The models land in the ./ollama folder next to the file. Do not add a ports: line. If your chat app runs in the same Compose file, it reaches Ollama at http://ollama:11434, and nobody else can. If you do publish the port, bind it to 127.0.0.1 so the network cannot see it. Why you should not open ports explains the risk. New to Compose? Read Docker Compose basics first.

Docker Desktop on macOS has no GPU passthrough. On a Mac, run Ollama natively instead.

Pull a model with docker compose exec ollama ollama pull llama3.1:8b, then pick it in the chat app's settings.

Option 2: pay a provider through an API key

This option needs no special hardware, because the heavy work happens at the provider. The machine only runs the chat app. LobeChat's Compose file starts six containers, so read its page before you pick a small server. You pay per token, so a quiet month is cheap and a busy one is not. Prices change often, so check the provider's pricing page on the day you decide.

Set a limit before you paste the key into an app. Providers differ:

• OpenRouter uses prepaid credits: you deposit money, and each request deducts from it. When the balance runs out, requests fail with an insufficient-credits error. You can also cap a single key with a credit limit.
• OpenAI recommends spend limits. An alert only sends a notification and traffic continues. A hard limit makes requests fail with a 429 error. Choose the hard limit.
• Anthropic lets you set your own monthly spend limit on the Billing page. Requests fail once you reach it.

Those are the rules as of October 2026. Screens and tiers change, so check the billing page of your provider.

Use one key per app. If an app leaks its key, you revoke that one key and nothing else breaks.

A leaked key, or a chat page open to strangers, lets other people spend your money. Every message a stranger types is billed to you. OpenPlate shows the problem clearly. Its Compose file can set a default endpoint for photo analysis, and the file's own comment says the key is public: every browser that loads the app can read it. The safer default is to leave it empty so each person brings their own key. Before you share any AI app, read before you share an app.

Privacy. With an API, your prompts and documents go to the provider. Read their data policy. OpenAI, for example, says API data is not used to train its models unless you opt in, and abuse-monitoring logs are kept for up to 30 days. OpenRouter says it does not log prompts unless you opt in, but the model provider behind it has its own policy.

The middle path

You do not have to choose once. AnythingLLM lets you set a different model for each workspace. A common setup:

• A workspace for private documents (tax papers, medical notes) that uses a small local model.
• A workspace for hard questions that uses an API model, with a spend limit.

This only works if you can run a small local model, so check the memory table above first.

Decide

• No GPU, and a small VPS or Pi? Use an API key. Set a hard spend limit first.
• A PC with an NVIDIA GPU and 8 GB or more of VRAM? Try Ollama with an 8B model. Keep the API as a backup.
• Private documents you will not send out? Use a local model for those, even if it is slower.
• Sharing the app with family or friends? Use one key per app with a low limit, and keep the page private. Use a private network, not an open page.
• Unsure? Start with an API key and a small prepaid budget of a few dollars. You can add Ollama later without changing the chat app.

Tip: if the app says the model is not found

Check that Ollama is running and that the model name matches exactly:

docker compose exec ollama ollama list

If the list is empty, pull the model again. If it shows the model, check the base URL in the app. Inside Compose it is http://ollama:11434, not localhost, because localhost inside the chat container means the chat container itself.