A ten-second wait followed by a wall of text feels broken. The same answer streamed word by word feels fast, even when total time is identical. Every AI product I ship streams, and Server-Sent Events over FastAPI is the simplest way to do it.

Why SSE over WebSockets

Token streaming is one-directional: the server talks, the client listens. SSE is plain HTTP, works through most proxies, reconnects automatically in the browser, and needs no upgrade handshake. WebSockets earn their complexity when the client also streams — live speech-to-text is the case where I use them instead.

The endpoint shape

  1. Accept the question and any document scope as a POST, validate, and run retrieval before opening the stream — retrieval errors should return a normal HTTP error, not a broken stream.
  2. Return a StreamingResponse with media type text/event-stream and an async generator that yields events.
  3. Send a first event carrying metadata: the citation map, the retrieval confidence, and a request id.
  4. Yield token events as the provider streams them; end with a done event that includes usage and timing.

Handling failure mid-stream

Once headers are sent you cannot change the status code, so errors must travel inside the stream as a typed event. RAG.NextUpgrad's multi-provider fallback works at this layer: if the primary provider fails before the first token, the generator switches providers and the client never notices; if it fails mid-answer, the client receives an error event and a retry hint rather than a truncated paragraph.

Pitfalls

  • Reverse proxies buffering the response — disable buffering for the stream route and send a comment line every few seconds as a heartbeat.
  • Idle timeouts on free hosts — heartbeats also keep the connection alive during a slow reranking pass.
  • Forgetting to flush — yield small strings; do not accumulate.
  • Client-side, use fetch with a reader rather than EventSource if you need POST bodies; EventSource only supports GET.

Streaming is a small amount of code with an outsized effect on how a product feels. It is also the first thing I demo to students who think an AI app is just an API call and a text box.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.