FastAPI Server-Sent Events: Streaming AI Chat Answers

FastAPI server-sent events streaming works by returning a long-lived HTTP response of type text/event-stream and yielding one small event per chunk of the model's answer. Since FastAPI 0.135.0 there's a built-in EventSourceResponse for this. Read the stream with fetch on the web and in Expo apps, turn off proxy buffering, and close streams that go silent.

TL;DR: Use SSE, not WebSockets, for AI chat: one request, one streamed answer, plain HTTP. On FastAPI 0.135.0 or later, return EventSourceResponse, which adds keep-alive pings and disables nginx buffering for you. Read it with fetch so you can send auth headers, and add an inactivity timeout and an explicit done event.

NoteTube streams its RAG chat this way: answers grounded in a student's own material arrive over server-sent events, with inactivity timeouts so dead connections don't hold resources. This post is the how-to: the event format, a complete endpoint, the client side, and the four things that break streaming in production.

Why use server-sent events instead of WebSockets?

Because an AI answer is one request followed by one stream of text from server to client, and SSE is built for exactly that shape over ordinary HTTP. WebSockets earn their extra machinery when both sides talk continuously, as in chat rooms, live collaboration, or market dashboards.

	Server-sent events	WebSockets
Direction	Server to client	Both ways
Protocol	Plain HTTP response	Upgraded connection
Proxies and CDNs	Usually fine, if buffering is off	Need explicit support
Auth	Normal HTTP headers (with fetch)	Usually a token at connect time
Best for	Streaming one answer, notifications	Continuous two-way traffic

We use both, for different jobs. The options trading platform we built fans market state to its dashboard through a WebSocket gateway, because that stream never ends and many screens listen to it. A chat answer starts, streams, and finishes. That's SSE.

What does an SSE event look like?

An SSE stream is UTF-8 text with the MIME type text/event-stream. Each event is one or more field lines, such as event: and data:, ended by a blank line, which tells the client to dispatch it. Lines starting with a colon are comments the client ignores (WHATWG HTML spec).

event: token
data: "Photosynthesis "

event: citation
data: {"source":"lecture-3.pdf","page":12}

: keep-alive

event: done
data: [DONE]

Name each event type so the client can tell tokens, citations, errors, and the end apart. The done event matters more than it looks. The spec says that if a stream ends before a blank line, pending data is discarded, and without an explicit end marker a client can't tell a finished answer from a dropped connection.

How do you build the endpoint in FastAPI?

On FastAPI 0.135.0 or later, set response_class=EventSourceResponse and write an async generator that yields ServerSentEvent objects. FastAPI JSON-encodes data, sends raw_data as-is, works with POST, sends a keep-alive ping every 15 seconds, and sets Cache-Control: no-cache and X-Accel-Buffering: no (FastAPI docs).

Here's a complete, runnable example. The answer_stream function stands in for retrieval plus your model provider's streaming client:

import asyncio
from collections.abc import AsyncIterable, AsyncIterator

from fastapi import FastAPI, Request
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel

app = FastAPI()
IDLE_TIMEOUT = 30  # seconds with no chunk before we close the stream


class ChatIn(BaseModel):
    question: str


async def answer_stream(question: str) -> AsyncIterator[dict]:
    # Stand-in for retrieval plus the provider's streaming API.
    for word in f"You asked: {question}".split():
        await asyncio.sleep(0.05)
        yield {"type": "token", "text": word + " "}


@app.post("/chat", response_class=EventSourceResponse)
async def chat(body: ChatIn, request: Request) -> AsyncIterable[ServerSentEvent]:
    chunks = answer_stream(body.question)
    try:
        while True:
            try:
                chunk = await asyncio.wait_for(anext(chunks), IDLE_TIMEOUT)
            except StopAsyncIteration:
                break
            except asyncio.TimeoutError:
                yield ServerSentEvent(data={"message": "timed out"}, event="error")
                return
            if await request.is_disconnected():
                return  # stop paying for tokens nobody will read
            yield ServerSentEvent(data=chunk["text"], event=chunk["type"])
        yield ServerSentEvent(raw_data="[DONE]", event="done")
    finally:
        await chunks.aclose()

On an older FastAPI, the same generator works with StreamingResponse(media_type="text/event-stream"), but you format the event: and data: lines yourself and set Cache-Control: no-cache and X-Accel-Buffering: no by hand. Add your auth dependency to the route as usual; the AI keys stay on this service, never in the client, as we explain in keeping AI API keys out of clients.

Everything inside the generator must be async. A blocking call there stalls every other stream on the process; our async FastAPI patterns post covers how to keep the event loop free.

How do you send auth headers with a stream?

Use fetch, not EventSource. The browser's EventSource constructor accepts a URL and a single option, withCredentials, so it can't send a POST body or an Authorization header (WHATWG HTML spec). With fetch, you POST the question with a bearer token and read the response body as a stream.

The same works in mobile apps. Expo's expo/fetch provides a WinterCG-compliant Fetch API that works across web and mobile, and its docs include a streaming example that reads resp.body with getReader() (Expo docs). Check that the Expo SDK you ship includes it before you rely on it. NoteTube's chat runs on a Next.js web app and an Expo app for iOS and Android, against the same AI service.

Four client-side mistakes to avoid

Most client bugs fall into four groups. Network chunks aren't events, so a parser that assumes they are breaks on the first slow connection. Re-rendering on every token makes phones stutter. Half-finished Markdown renders strangely. And clearing the screen on an error throws away the part of the answer the user already read.

1. Buffer and split. A chunk can hold half an event or three. Append incoming text to a buffer and split on the blank line before parsing.
2. Batch renders, for example once per animation frame, instead of once per token.
3. Render Markdown carefully. Use plain text until done, or a renderer that tolerates an open code block.
4. Keep the partial answer on failure, with a clear retry button.

Citations deserve their own rule. Show them as they arrive, but make them tappable only after done, so a student doesn't open a source that's about to be replaced by a better match. In a study product like NoteTube, the citation is the reason to trust the answer, so it's worth getting right.

Why does the whole answer arrive at once in production?

Something between the server and the user is buffering. It's the most common streaming bug: tokens flow on localhost, and in production the answer appears all at once after ten seconds. Check every hop, starting with the reverse proxy.

1. Reverse proxies. nginx has proxy_buffering on by default, and a response header of X-Accel-Buffering: no turns buffering off for that response (nginx docs). The native FastAPI response sets it for you; a hand-rolled one must.
2. Compression middleware can hold output until it has enough to compress. Exclude event streams.
3. Load balancers and CDNs may buffer or time out long responses. Check their streaming settings.
4. Serverless platforms may cap response length or not stream at all.

How do you stop dead streams from holding resources?

With two timers and a disconnect check. A stream can die without closing: a phone loses signal, a laptop sleeps, a proxy drops the connection quietly. If the server keeps the generator running, it holds a model request and memory for nobody.

First, an inactivity timeout: if no chunk arrives from the model for a set period, close the stream with an error event, as the example does. NoteTube's streaming chat uses inactivity timeouts for this reason. Don't add a total-duration limit that cuts off a long answer that's still producing tokens.

Second, keep-alive comments. The spec warns that legacy proxies drop HTTP connections after a short timeout and suggests a comment line about every 15 seconds; the native FastAPI response does this automatically. Third, check request.is_disconnected() between chunks and stop, closing the upstream generator so the provider request is cancelled too, if your client library supports it.

Errors and resumes mid-stream

Once the first byte is sent, the HTTP status is fixed at 200. So send errors as events, event: error with a short message, then end the stream. The client shows the partial answer, marks it as cut short, and offers a retry.

For long answers on mobile, consider resumable streams. Give each answer an ID and keep generated chunks on the server for a short time. The browser's EventSource sends a Last-Event-ID header when it reconnects, and with fetch you can send the last ID yourself, so the server can replay what was missed instead of paying for the whole answer twice.

Plan capacity by open streams

One async FastAPI process holds more streams than you'd expect, because each stream spends most of its life waiting on the model. Capacity planning is about concurrent streams, not requests per second. Put a limit on concurrent model calls so a spike doesn't exhaust your provider quota, and watch open connections alongside latency.

FAQ

Which FastAPI version has built-in SSE support?

FastAPI 0.135.0 added EventSourceResponse and ServerSentEvent in fastapi.sse. They handle JSON encoding, keep-alive pings every 15 seconds, and the no-cache and no-buffering headers. On older versions, use StreamingResponse with the text/event-stream media type and write those parts yourself. Pin the version in your requirements so a deploy doesn't silently fall back.

Can SSE stream a response to a POST request?

Yes. SSE is a response format, so any method works, and FastAPI's docs say so explicitly. Only the browser's EventSource is limited to GET. Most AI chat clients POST the question with fetch, then read the streamed body, which also lets them send an Authorization header.

Should streamed tokens be saved to the database?

Save the finished answer, not every token. Write once on done, with the citations and the model used. If you support resuming long answers, keep chunks in a short-lived cache keyed by answer ID instead, and let them expire. Writing every token to Postgres adds load and buys nothing.

Adding streaming to an AI feature or a realtime product? See our FastAPI realtime work or our AI chatbot development service, and send us the endpoint that isn't streaming the way it should.