Embeddings and Vector Databases
Why a small chat model needs a memory it can check, how embeddings place similar meanings close together, why dedicated embedding models beat chat models for retrieval, and what vector databases like Chroma add: storage, nearest-neighbor search, and approximate indexes.
The Problem: Confident and Wrong
Ask Module 8’s chatbot about a movie and it answers fluently — whether or not it knows anything about that movie. A small model’s knowledge is whatever it absorbed during training: incomplete, frozen at a cutoff date, and blind to your data. Asked about something it doesn’t know, it often hallucinates: it produces plausible text that isn’t true.
The fix is to stop asking the model to remember and start giving it the facts each time — retrieve relevant information from your own data and put it in the prompt. That’s retrieval-augmented generation (RAG). Before building the pipeline in the next lesson, we need the two technologies that make the retrieval step work: embeddings and vector databases.
Embeddings: Meaning as Coordinates
In Module 6, a movie became a vector of word counts. Counts capture spelling, not meaning: “a heist in a dream” and “thieves infiltrate the subconscious” share almost no words.
An embedding is a vector — typically hundreds or thousands of numbers long — produced by a neural network trained so that texts with similar meanings get nearby vectors. The individual numbers aren’t interpretable on their own; what matters is distance. In that space, “kitten” sits near “cat” and “feline,” and a plot about a dream heist sits near other mind-bending thrillers even with no words in common.
This is what makes semantic search possible: embed the user’s question with the same model, then find the stored vectors closest to it. Closeness is measured with the same tools as Module 6 — cosine similarity, the dot product, or Euclidean distance.
Choosing an embedding model
Embeddings come from an embedding model. The source project computes them with the same small chat model it uses for answers (gemma2:2b, through Ollama). That works, but it isn’t ideal: chat models are trained to generate text, not to produce vectors that rank well in search. Dedicated embedding models — nomic-embed-text is one that runs in Ollama — are smaller, faster, and usually retrieve better. The source project itself suggests switching to one when the chat model struggles.
One rule is non-negotiable: embed the documents and the queries with the same model. Vectors from different models live in different spaces, and comparing them is meaningless.
Vector Databases
A vector database stores embeddings and finds the nearest ones quickly. Each record typically holds:
- the vector;
- the original text it came from, so it can be handed to the LLM;
- metadata — an ID, a source, a page number — for filtering and citations.
| Capability | Why it matters |
|---|---|
| Vector storage | Persist millions of high-dimensional vectors efficiently |
| Similarity search | Return the k nearest neighbors to a query vector |
| High-dimensional indexing | Traditional database indexes don’t help with “closest in 768 dimensions” |
| Metadata filtering | ”Nearest neighbors, but only from this source or date range” |
| Low latency and scale | Fast enough for interactive apps; many scale horizontally |
Exact vs. approximate search
Comparing a query with every stored vector — exact nearest-neighbor search — is perfectly accurate but slows down as the collection grows. Most vector databases build approximate nearest neighbor (ANN) indexes — graph-based structures such as HNSW are common — that find almost the closest results in a tiny fraction of the time. For retrieval, trading a sliver of accuracy for a large speed-up is almost always worth it.
The landscape
| Option | Type |
|---|---|
| Chroma | Open-source; runs embedded in your app and persists to a local folder — used in this course |
| Pinecone | Fully managed cloud service |
| Milvus, Weaviate, Qdrant | Open-source vector databases built for scale; self-host or use managed versions |
| Faiss | A similarity-search library rather than a database — fast building block, you handle storage |
Vector search also powers recommendation systems, image and video similarity, semantic search engines, and anomaly detection — anywhere “find things like this” matters more than exact matching.
Common mistake: upgrading to a better embedding model for queries while leaving the stored vectors as they were.
The stored vectors were made by the old model; the query vector by the new one. They’re in different spaces, so similarity scores become noise and retrieval quietly returns irrelevant passages — no error, just bad answers. When the embedding model changes, re-embed and re-index everything, and record which model built each index. Also remember that a vector database holds your data in another form: protect it with the same access controls as the source documents.
1. What property makes embeddings useful for retrieval?
Embedding models are trained so that distance reflects meaning, enabling semantic search.
2. Why must documents and queries be embedded with the same model?
Distances only mean something within one model's space.
3. What does an approximate nearest neighbor (ANN) index trade?
ANN indexes find nearly the closest results far faster than comparing against every vector.
4. Which item is a similarity-search library rather than a database?
Faiss provides fast similarity search; storage, persistence, and serving are up to you.
Up Next
With embeddings and a vector store in hand, the next lesson assembles the full RAG pipeline — loading, splitting, embedding, retrieving, prompting, and generating — and tests whether it actually stops the model from making things up.