API Gateways, Ports, and Streaming
Module 8 · Adding an LLM

API Gateways, Ports, and Streaming

Why the system funnels every request through one gateway, its pros and cons, a full map of the six tiers with their ports, protocols, and connections, the loopback interface, and how LLM responses stream to the screen with chunked transfer encoding.

From Many Addresses to One

In Module 6 the frontend knew two backend addresses — one for search, one for recommendations. Add an LLM server and an MLflow server, and a frontend that talks to each service directly would need to know, configure, authenticate with, and handle failures from four or five of them.

The system is restructured so Express is the only endpoint the frontend ever calls. Express forwards each request to the right service behind it: the database for search, the ML service for recommendations, the LLM server for chat. In this role Express is both a reverse proxy (it forwards requests on behalf of clients) and an API gateway (a single, managed entry point to a set of backend services).

Direct callsFrontendSearch APIML serviceLLM serverMLflow4 public endpoints · 4 addresses to configureAPI gatewayFrontendGatewayExpressinternal onlyDatabaseML serviceLLM serverMLflow1 public endpoint · services stay private

Pros and cons of a gateway

ProsCons
Simpler frontend — one address to configureSingle point of failure — if the gateway goes down, everything does
Centralized routing — one place decides where traffic goesAdded latency — every request takes an extra hop
Independent scaling — services behind it can scale and move freelyBottleneck risk — heavy traffic can overwhelm it if it isn’t scaled
Security at one entry point — authentication, rate limiting, logging, input validationGrowing complexity — the gateway accumulates responsibilities and gets harder to maintain

Production systems mitigate the cons by running several gateway instances behind a load balancer, keeping the gateway thin (routing and cross-cutting concerns only), and often using a dedicated gateway product instead of hand-written code.


Mapping the System: Tiers, Ports, Protocols

A useful exercise with any architecture is to list every running component, its port, and every connection between them. The movie app at this stage has six tiers:

#TierDefault port
1LLM server (Ollama)11434
2Database (MongoDB Atlas)27017
3Application server / gateway (Express)3000
4MLflow tracking server5000
5ML API server (FastAPI)8000
6Mobile app (Expo, web version)8081

And seven connections:

  • From Express: to MongoDB (mongodb+srv), to Ollama, and to FastAPI.
  • From FastAPI: to Express (for movie data), to Ollama, and to MLflow (to load the model).
  • From the frontend: to Express only.

Only two protocol types appear: HTTP (every service-to-service call) and mongodb+srv (the database driver’s protocol). That’s typical — most internal traffic in modern systems is plain HTTP carrying JSON.

Loopback: localhost and 127.0.0.1

Most of these connections target 127.0.0.1 — the loopback address, also called localhost. Traffic to it never leaves the machine. A service that listens only on loopback can’t be reached from the network at all, even if a firewall rule is wrong. That’s the right default for Ollama, FastAPI, and MLflow here: only the gateway needs a public face. (Development workspaces complicate this by forwarding ports — another reason to check each port’s visibility.)


Streaming Responses

LLMs generate text one piece at a time, and a long answer can take many seconds. Two ways to deliver it:

  • Wait for the whole thing — the user stares at a spinner, then the full answer appears.
  • Stream it — send each piece as soon as it’s generated, so text appears progressively.

Total time is the same; what changes is time to first token — how long before the user sees anything. Streaming makes a slow model feel responsive.

How streaming travels over HTTP

Streaming means continuous transmission where the receiver can process data as it arrives. Over HTTP/1.1, the standard mechanism is chunked transfer encoding: the server sends the header Transfer-Encoding: chunked, then writes the body as a series of chunks, each prefixed with its size in bytes, and signals the end with a zero-length chunk. It’s built for exactly this situation — when the total length isn’t known in advance.

In this module, the chat path really streams end to end:

  1. Express calls Ollama’s chat API with streaming enabled.
  2. As each piece of the answer arrives from Ollama, Express writes it straight through to the frontend’s response.
  3. The frontend reads the response body as a stream, decodes each chunk of bytes into text, and appends it to the chat bubble.

Other streaming mechanisms you’ll meet include Server-Sent Events (a simple standard for server-to-browser streams of messages) and WebSockets (two-way, long-lived connections).

Two kinds of “chunking.” Here, a chunk is a piece of an HTTP response. In Module 9, chunking means splitting documents into passages before indexing them. Same word, unrelated ideas.

⚠️ Common Pitfall: A Gateway That Forwards Anything

Common mistake: writing a “generic” proxy route that takes a target path or URL from the client and forwards the request.

That turns your gateway into a tool for attackers to reach internal services directly — or, if it accepts full URLs, to make your server fetch arbitrary addresses (server-side request forgery, SSRF). A gateway should expose a fixed allowlist of routes, validate inputs, and apply rate limits — especially on the chat route, where every request costs real compute.

🧠 Knowledge Check

1. Which is a genuine disadvantage of routing every request through one gateway?

2. Why bind the FastAPI and Ollama servers to 127.0.0.1?

3. What does streaming an LLM response improve?

4. In chunked transfer encoding, how does the receiver know the response is finished?


Up Next

The chatbot works — but a small model answering from memory makes things up. Module 9 gives it a memory it can check: embeddings, a vector database, and retrieval-augmented generation.