API Gateways, Ports, and Streaming
Why the system funnels every request through one gateway, its pros and cons, a full map of the six tiers with their ports, protocols, and connections, the loopback interface, and how LLM responses stream to the screen with chunked transfer encoding.
From Many Addresses to One
In Module 6 the frontend knew two backend addresses — one for search, one for recommendations. Add an LLM server and an MLflow server, and a frontend that talks to each service directly would need to know, configure, authenticate with, and handle failures from four or five of them.
The system is restructured so Express is the only endpoint the frontend ever calls. Express forwards each request to the right service behind it: the database for search, the ML service for recommendations, the LLM server for chat. In this role Express is both a reverse proxy (it forwards requests on behalf of clients) and an API gateway (a single, managed entry point to a set of backend services).
Pros and cons of a gateway
| Pros | Cons |
|---|---|
| Simpler frontend — one address to configure | Single point of failure — if the gateway goes down, everything does |
| Centralized routing — one place decides where traffic goes | Added latency — every request takes an extra hop |
| Independent scaling — services behind it can scale and move freely | Bottleneck risk — heavy traffic can overwhelm it if it isn’t scaled |
| Security at one entry point — authentication, rate limiting, logging, input validation | Growing complexity — the gateway accumulates responsibilities and gets harder to maintain |
Production systems mitigate the cons by running several gateway instances behind a load balancer, keeping the gateway thin (routing and cross-cutting concerns only), and often using a dedicated gateway product instead of hand-written code.
Mapping the System: Tiers, Ports, Protocols
A useful exercise with any architecture is to list every running component, its port, and every connection between them. The movie app at this stage has six tiers:
| # | Tier | Default port |
|---|---|---|
| 1 | LLM server (Ollama) | 11434 |
| 2 | Database (MongoDB Atlas) | 27017 |
| 3 | Application server / gateway (Express) | 3000 |
| 4 | MLflow tracking server | 5000 |
| 5 | ML API server (FastAPI) | 8000 |
| 6 | Mobile app (Expo, web version) | 8081 |
And seven connections:
- From Express: to MongoDB (
mongodb+srv), to Ollama, and to FastAPI. - From FastAPI: to Express (for movie data), to Ollama, and to MLflow (to load the model).
- From the frontend: to Express only.
Only two protocol types appear: HTTP (every service-to-service call) and mongodb+srv (the database driver’s protocol). That’s typical — most internal traffic in modern systems is plain HTTP carrying JSON.
Loopback: localhost and 127.0.0.1
Most of these connections target 127.0.0.1 — the loopback address, also called localhost. Traffic to it never leaves the machine. A service that listens only on loopback can’t be reached from the network at all, even if a firewall rule is wrong. That’s the right default for Ollama, FastAPI, and MLflow here: only the gateway needs a public face. (Development workspaces complicate this by forwarding ports — another reason to check each port’s visibility.)
Streaming Responses
LLMs generate text one piece at a time, and a long answer can take many seconds. Two ways to deliver it:
- Wait for the whole thing — the user stares at a spinner, then the full answer appears.
- Stream it — send each piece as soon as it’s generated, so text appears progressively.
Total time is the same; what changes is time to first token — how long before the user sees anything. Streaming makes a slow model feel responsive.
How streaming travels over HTTP
Streaming means continuous transmission where the receiver can process data as it arrives. Over HTTP/1.1, the standard mechanism is chunked transfer encoding: the server sends the header Transfer-Encoding: chunked, then writes the body as a series of chunks, each prefixed with its size in bytes, and signals the end with a zero-length chunk. It’s built for exactly this situation — when the total length isn’t known in advance.
In this module, the chat path really streams end to end:
- Express calls Ollama’s chat API with streaming enabled.
- As each piece of the answer arrives from Ollama, Express writes it straight through to the frontend’s response.
- The frontend reads the response body as a stream, decodes each chunk of bytes into text, and appends it to the chat bubble.
Other streaming mechanisms you’ll meet include Server-Sent Events (a simple standard for server-to-browser streams of messages) and WebSockets (two-way, long-lived connections).
Two kinds of “chunking.” Here, a chunk is a piece of an HTTP response. In Module 9, chunking means splitting documents into passages before indexing them. Same word, unrelated ideas.
Common mistake: writing a “generic” proxy route that takes a target path or URL from the client and forwards the request.
That turns your gateway into a tool for attackers to reach internal services directly — or, if it accepts full URLs, to make your server fetch arbitrary addresses (server-side request forgery, SSRF). A gateway should expose a fixed allowlist of routes, validate inputs, and apply rate limits — especially on the chat route, where every request costs real compute.
1. Which is a genuine disadvantage of routing every request through one gateway?
Centralizing traffic means the gateway must be made highly available and scaled — otherwise it limits the whole system.
2. Why bind the FastAPI and Ollama servers to 127.0.0.1?
Listening only on loopback removes these services from the network entirely, shrinking attack surface.
3. What does streaming an LLM response improve?
Streaming doesn't make generation faster; it makes the wait visible and progressive.
4. In chunked transfer encoding, how does the receiver know the response is finished?
Each chunk carries its size; a chunk of size zero marks the end.
Up Next
The chatbot works — but a small model answering from memory makes things up. Module 9 gives it a memory it can check: embeddings, a vector database, and retrieval-augmented generation.