Running an LLM on Your Own Server
Module 8 · Adding an LLM

Running an LLM on Your Own Server

The chat user story brings a large language model into the system. Learn what model parameters are, how model size drives memory and quality, what Ollama does, how two small open models — Gemma 2 and TinyLlama — compare, and the trade-offs of self-hosting vs. calling a hosted API.

The Third User Story

As a movie enthusiast using a mobile device, I want to chat and get movie recommendations from my selections.

Search filtered data. Recommendations computed similarity. Chat requires something new: understanding a free-form question and generating a natural-language answer. That’s the job of a large language model (LLM).

all calls query fetch movies recommend chat load model log runs GitHub repo Codespaces dev env GitHub Actions test · ZAP scan · deploy MongoDB Atlas document database LLM server Ollama · Gemma / TinyLlama NEW User Mobile + web app React Native Web API gateway Express · single entry CHANGED runs in Codespaces ML service FastAPI · model server Training scripts LR · RF · GB · NN MLflow tracking runs · metrics · models
Module 8: an LLM server joins the system, and Express becomes the single gateway every frontend request passes through. Highlighted in red: what's new or changed in this module.

What an LLM Is, Briefly

An LLM is a neural network trained on enormous amounts of text to predict the next piece of text given everything before it. Generate one piece, append it, predict again — repeat, and you get sentences, answers, and summaries.

  • Input: a text string — a question, an instruction, or a document to summarize.
  • Output: generated text — an answer, a summary, a recommendation.

Module 10 looks more closely at tokens, embeddings, and how these models are built. For now, the important ideas are size and where the model runs.

Parameters: What “2 Billion” Means

Parameters are the values a model learns from its training data. In neural networks they’re mainly:

  • Weights — the strength of each connection between neurons in neighboring layers;
  • Biases — constants added to each layer’s inputs that shift when neurons activate.

During training, the model makes predictions, measures how wrong they are (the loss), and nudges every parameter to reduce that loss, using an optimization method such as gradient descent. Repeat over trillions of words and the parameters come to encode grammar, facts, and reasoning patterns.

The parameter count is the headline measure of model size. More parameters can capture subtler patterns, but cost more memory and compute — and need more data to train well.

Size drives memory

A model’s parameters must fit in memory to run. A useful rule of thumb:

memory ≈ number of parameters × bytes per parameter

At 16-bit precision (2 bytes each), a 2-billion-parameter model needs about 4 GB just for its weights. Quantization stores weights at lower precision — commonly 4 bits — cutting that by roughly three-quarters, with a modest loss in quality. That’s how small models run on laptops, cloud dev environments, and even phones.


Ollama: A Local LLM Server

Ollama is a lightweight tool for downloading and running open language models on your own machine. It provides:

  • a model library — pull a model by name and tag, such as gemma2:2b or tinyllama;
  • an interactive run mode for chatting in the terminal;
  • a background server with an HTTP API (on port 11434 by default) that applications call to generate text, with responses that can stream token by token.

In the movie app, Ollama is the LLM server tier. It can run a variety of open model families — Llama, Mistral, Gemma, and others — and swapping models is a matter of pulling a different one and changing a name in configuration.

Two Small Open Models

Open-weight models publish their trained parameters, so anyone can download and run them. The movie app uses one of two:

Gemma 2 (2B)TinyLlama
FromGoogle DeepMind, built on the research behind the Gemini modelsAn open-source project using the Llama architecture
Size~2 billion parameters (family also has 9B and 27B)1.1 billion parameters
Training dataAbout 2 trillion tokens for the 2B model (13T for 27B) — web documents, code, and mathematics, primarily EnglishAbout 3 trillion tokens
Download (quantized, via Ollama)Roughly 1.6 GBRoughly 0.6 GB
Good forBetter answers on modest hardwareVery constrained machines — the fallback when memory runs out

Gemma’s training data was filtered for safety: harmful and illegal content was removed, and automated techniques screened out personal and sensitive information. Data filtering is part of what “responsible” model development means — a model can only be as clean as what it learned from.

Both models are tiny compared with frontier models that have hundreds of billions of parameters or more. They’re fast and cheap, but they know less, reason less reliably, and hallucinate — produce confident, fluent, wrong answers — more often. Keep that in mind; Module 9 exists largely to manage it.

These specific models will date quickly — newer generations appear every few months. The concepts don’t: parameter count, quantization, memory budget, and the quality–cost trade-off apply to whatever model you pick next.


Self-Hosting vs. a Hosted API

Self-hosted (Ollama + open model)Hosted API (a provider’s frontier model)
DataPrompts never leave your infrastructurePrompts go to the provider, under its data terms
CostYour hardware; no per-request feePay per token; no hardware
CapabilityLimited by what your hardware can runAccess to the most capable models
OperationsYou install, update, scale, and secure itProvider handles it
AvailabilityWorks offline and with no vendor dependencyDepends on the provider’s uptime and limits

Neither choice is universally right. Sensitive data, offline use, and predictable cost favor self-hosting; capability and zero operations favor a hosted API. Many systems route simple requests to a small local model and hard ones to a hosted model.

⚠️ Common Pitfall: Exposing the LLM Server Directly

Common mistake: making the LLM server’s port public so the frontend can call it.

A local model server typically has no authentication. Anyone who finds the port can run unlimited prompts on your hardware, pull or delete models, and bypass every rule your application applies. Keep the LLM server reachable only from your own backend (bound to localhost or a private network), and put the application tier in front of it — which is exactly what the next lesson does.

🧠 Knowledge Check

1. Roughly how much memory do the weights of a 2-billion-parameter model need at 16-bit precision?

2. What are a neural network's parameters?

3. Why might a team choose TinyLlama over Gemma 2 2B?

4. Which is the strongest reason to self-host an open model rather than call a hosted API?


Up Next

The system now has five services. The next lesson reorganizes them behind a single API gateway, maps every port and protocol in the system, and explains how chat responses stream to the screen — and when “streaming” is only an illusion.