Embeddings and Vector Databases
Module 9 · Retrieval-Augmented Generation

Embeddings and Vector Databases

Why a small chat model needs a memory it can check, how embeddings place similar meanings close together, why dedicated embedding models beat chat models for retrieval, and what vector databases like Chroma add: storage, nearest-neighbor search, and approximate indexes.

The Problem: Confident and Wrong

Ask Module 8’s chatbot about a movie and it answers fluently — whether or not it knows anything about that movie. A small model’s knowledge is whatever it absorbed during training: incomplete, frozen at a cutoff date, and blind to your data. Asked about something it doesn’t know, it often hallucinates: it produces plausible text that isn’t true.

The fix is to stop asking the model to remember and start giving it the facts each time — retrieve relevant information from your own data and put it in the prompt. That’s retrieval-augmented generation (RAG). Before building the pipeline in the next lesson, we need the two technologies that make the retrieval step work: embeddings and vector databases.

all calls query fetch movies recommend · chat generate load model log runs retrieve store GitHub repo Codespaces dev env GitHub Actions test · ZAP scan · deploy MongoDB Atlas document database LLM server Ollama · Gemma / TinyLlama User Mobile + web app React Native Web API gateway Express · single entry runs in Codespaces ML service FastAPI · recommend + RAG CHANGED Training scripts LR · RF · GB · NN MLflow tracking runs · metrics · models Vector DB ChromaDB NEW Ingest script LangChain · movies NEW
Module 9: an ingest script loads movie descriptions into a vector database, and the ML service retrieves from it before asking the LLM. Highlighted in red: what's new or changed in this module.

Embeddings: Meaning as Coordinates

In Module 6, a movie became a vector of word counts. Counts capture spelling, not meaning: “a heist in a dream” and “thieves infiltrate the subconscious” share almost no words.

An embedding is a vector — typically hundreds or thousands of numbers long — produced by a neural network trained so that texts with similar meanings get nearby vectors. The individual numbers aren’t interpretable on their own; what matters is distance. In that space, “kitten” sits near “cat” and “feline,” and a plot about a dream heist sits near other mind-bending thrillers even with no words in common.

embedding space (simplified to 2 of many dimensions)space explorationromantic comedycrime & heists◆ question: “a team steals secrets from dreams?“nearest neighbors= most relevant passages

This is what makes semantic search possible: embed the user’s question with the same model, then find the stored vectors closest to it. Closeness is measured with the same tools as Module 6 — cosine similarity, the dot product, or Euclidean distance.

Choosing an embedding model

Embeddings come from an embedding model. The source project computes them with the same small chat model it uses for answers (gemma2:2b, through Ollama). That works, but it isn’t ideal: chat models are trained to generate text, not to produce vectors that rank well in search. Dedicated embedding models — nomic-embed-text is one that runs in Ollama — are smaller, faster, and usually retrieve better. The source project itself suggests switching to one when the chat model struggles.

One rule is non-negotiable: embed the documents and the queries with the same model. Vectors from different models live in different spaces, and comparing them is meaningless.


Vector Databases

A vector database stores embeddings and finds the nearest ones quickly. Each record typically holds:

  • the vector;
  • the original text it came from, so it can be handed to the LLM;
  • metadata — an ID, a source, a page number — for filtering and citations.
CapabilityWhy it matters
Vector storagePersist millions of high-dimensional vectors efficiently
Similarity searchReturn the k nearest neighbors to a query vector
High-dimensional indexingTraditional database indexes don’t help with “closest in 768 dimensions”
Metadata filtering”Nearest neighbors, but only from this source or date range”
Low latency and scaleFast enough for interactive apps; many scale horizontally

Comparing a query with every stored vector — exact nearest-neighbor search — is perfectly accurate but slows down as the collection grows. Most vector databases build approximate nearest neighbor (ANN) indexes — graph-based structures such as HNSW are common — that find almost the closest results in a tiny fraction of the time. For retrieval, trading a sliver of accuracy for a large speed-up is almost always worth it.

The landscape

OptionType
ChromaOpen-source; runs embedded in your app and persists to a local folder — used in this course
PineconeFully managed cloud service
Milvus, Weaviate, QdrantOpen-source vector databases built for scale; self-host or use managed versions
FaissA similarity-search library rather than a database — fast building block, you handle storage

Vector search also powers recommendation systems, image and video similarity, semantic search engines, and anomaly detection — anywhere “find things like this” matters more than exact matching.

⚠️ Common Pitfall: Changing the Embedding Model Without Re-Indexing

Common mistake: upgrading to a better embedding model for queries while leaving the stored vectors as they were.

The stored vectors were made by the old model; the query vector by the new one. They’re in different spaces, so similarity scores become noise and retrieval quietly returns irrelevant passages — no error, just bad answers. When the embedding model changes, re-embed and re-index everything, and record which model built each index. Also remember that a vector database holds your data in another form: protect it with the same access controls as the source documents.

🧠 Knowledge Check

1. What property makes embeddings useful for retrieval?

2. Why must documents and queries be embedded with the same model?

3. What does an approximate nearest neighbor (ANN) index trade?

4. Which item is a similarity-search library rather than a database?


Up Next

With embeddings and a vector store in hand, the next lesson assembles the full RAG pipeline — loading, splitting, embedding, retrieving, prompting, and generating — and tests whether it actually stops the model from making things up.