Streaming LLM Responses End to End: Tokens, Backpressure, and Honest Loading States

You will build a complete token-streaming pipeline: a server route that forwards LLM deltas as server-sent events, a client that reads it with the standard ReadableStream API, and the behaviour that makes it reliable — backpressure, aborts, partial tool-call handling, retry limits, proxy fixes, and loading states that do not lie.

How Token Streaming Actually Works

An LLM generates tokens sequentially, and most provider SDKs expose that sequence as an async iterator. Your job is to move each delta to the browser without buffering it, corrupting it, or losing track of whether the user is still there.

Keep three layers separate: provider to server (normalise text, tool calls, finish reasons, and errors into one format), server to browser (choose a transport and respect flow control, so a slow client does not force you to hold a whole response in memory), and browser to UI (decode incrementally, parse complete events, render partial state honestly).

Choosing a Transport: SSE, WebSockets, or Chunked Fetch

SSE and chunked fetch are similar on the wire: one HTTP response delivered in pieces. SSE adds a standard format — event: and data: fields separated by a blank line. Chunked fetch leaves the format to you. Use WebSockets only when the client must talk back on the same channel during generation; otherwise a socket adds operational weight without adding capability. This article uses SSE over fetch, which supports POST bodies and Authorization headers in a debuggable standard format.

The Server: Streaming From an Async Iterator

This route uses only standard web APIs: Request, Response, ReadableStream, and TextEncoder. Adapt the delta mapping to your provider; keep the mechanics as shown.

type ModelDelta =
  | { type: "text"; text: string }
  | { type: "tool-call-start"; id: string; name: string }
  | { type: "tool-call-delta"; id: string; argumentsText: string }
  | { type: "finish"; reason: string };
 
declare function streamModel(input: {
  prompt: string;
  signal: AbortSignal;
}): AsyncIterable<ModelDelta>;
 
function encodeEvent(encoder: TextEncoder, event: string, data: unknown): Uint8Array {
  return encoder.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);
}
 
export async function POST(request: Request): Promise<Response> {
  const { prompt } = (await request.json()) as { prompt: string };
  const encoder = new TextEncoder();
 
  const stream = new ReadableStream<Uint8Array>({
    async start(controller) {
      try {
        for await (const delta of streamModel({ prompt, signal: request.signal })) {
          if (request.signal.aborted) break;
          controller.enqueue(encodeEvent(encoder, delta.type, delta));
        }
        controller.enqueue(encodeEvent(encoder, "done", { ok: true }));
      } catch (error) {
        if (request.signal.aborted) return;
        const message = error instanceof Error ? error.message : "Stream failed";
        controller.enqueue(encodeEvent(encoder, "error", { message }));
      } finally {
        controller.close();
      }
    },
    cancel() {
      // Runtime cancelled the stream. request.signal propagation
      // stops the upstream call; add provider teardown here if needed.
    },
  });
 
  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
      "X-Accel-Buffering": "no",
    },
  });
}

Every event ends with a blank line — the client parser depends on it. Errors after the response starts cannot change the HTTP status, so send them in-band as an error event. Pass request.signal upstream, or a closed tab leaves you paying for a generation nobody will read.

Handling Backpressure Correctly

Backpressure is the signal that the consumer is slower than the producer. Over a constrained connection, it is the difference between gentle pressure and a server holding a long answer in memory per client.

Inside a ReadableStream source, controller.desiredSize shows whether the queue wants more data: positive means room, zero or negative means wait. You cannot pause a model precisely, but you can stop pulling from its iterator until the queue drains by structuring the source around pull() — reading the next delta only when pull() is called — instead of an eager loop in start().

An eager start() loop is acceptable for short responses, but you have deferred flow control to the queue, not implemented it. For long responses, slow clients, or many concurrent streams, use pull(). Never drop text deltas to shed load — a dropped token corrupts the answer silently. Close with an explicit error event instead.

On the client, reader.read() only delivers when you ask, so your render loop sets the pace. Append incrementally and batch rendering to animation frames, rather than re-rendering the whole document per delta.

The Client: Reading a ReadableStream

Use fetch rather than EventSource when you need POST or custom headers; EventSource only issues GET requests. A chunk can contain half an event, one event, or several, and none of those boundaries align with token boundaries — so your parser must keep a leftover buffer.

export type StreamEvent =
  | { type: "text"; text: string }
  | { type: "tool-call-start"; id: string; name: string }
  | { type: "tool-call-delta"; id: string; argumentsText: string }
  | { type: "finish"; reason: string };
 
function parseSseBlock(block: string): { event: string; data: string } | null {
  let event = "message";
  const dataLines: string[] = [];
  for (const line of block.split("\n")) {
    if (line.startsWith("event:")) event = line.slice("event:".length).trim();
    else if (line.startsWith("data:")) dataLines.push(line.slice("data:".length).trim());
  }
  return dataLines.length ? { event, data: dataLines.join("\n") } : null;
}
 
export async function readChatStream(
  response: Response,
  onEvent: (event: StreamEvent) => void,
): Promise<void> {
  if (!response.ok) throw new Error(`Request failed: ${response.status}`);
  if (!response.body) throw new Error("Response has no body to stream");
 
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";
 
  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      buffer += decoder.decode(value, { stream: true });
      let boundary = buffer.indexOf("\n\n");
      while (boundary !== -1) {
        const parsed = parseSseBlock(buffer.slice(0, boundary));
        buffer = buffer.slice(boundary + 2);
        if (parsed?.event === "error") {
          throw new Error((JSON.parse(parsed.data) as { message: string }).message);
        }
        if (parsed && parsed.event !== "done") {
          onEvent({ ...(JSON.parse(parsed.data) as object), type: parsed.event } as StreamEvent);
        }
        boundary = buffer.indexOf("\n\n");
      }
    }
  } finally {
    reader.releaseLock();
  }
}

Always decode with { stream: true }: a multi-byte character can be split across chunks, and decoding without it produces replacement characters at boundaries.

Client Disconnects, Aborts, and Cleanup

Distinguish three endings. Normal completion: the server sends a finish event and closes; finalise the message and re-enable input. User abort: the user presses stop or navigates away; abort the per-request AbortController, which cancels the fetch, closes the server connection, and should abort the upstream call via request.signal. Failure: the network drops or the server sends an in-band error; keep the partial text, mark it incomplete, and offer a retry — do not present it as a finished answer.

Create one AbortController per request, pass its signal to fetch, and clear it when the stream settles. Do not enqueue after cancellation on the server; writes to a cancelled stream are discarded or throw, and logging that noise as a model error pollutes reporting. Tie any heartbeat or typing indicator to the same controller, so nothing outlives the stream it describes.

Partial JSON and Tool-Call Deltas

Text deltas are forgiving: append them in order and the result reads correctly. Tool calls are not. A model emits a start event with an identifier and name, then argument fragments that are pieces of a JSON string — invalid individually, valid concatenated in full.

Accumulate fragments in a map keyed by the call identifier. Only call JSON.parse when the provider signals completion — a finish event for that call, a new call starting, or the stream ending. Do not execute a tool, render a form from guessed values, or trigger a side effect from a partial parse unless the action is explicitly speculative and reversible. If you want to show progress, show that a named tool is being prepared and its arguments are still arriving. That is accurate; guessed argument values are not.

export class ToolCallAccumulator {
  private pending = new Map<string, { name: string; argumentsText: string }>();
 
  start(id: string, name: string): void {
    this.pending.set(id, { name, argumentsText: "" });
  }
 
  delta(id: string, argumentsText: string): void {
    const call = this.pending.get(id);
    if (!call) throw new Error(`Delta for unknown tool call: ${id}`);
    call.argumentsText += argumentsText;
  }
 
  complete(id: string): { name: string; args: Record<string, unknown> } {
    const call = this.pending.get(id);
    if (!call) throw new Error(`Unknown tool call: ${id}`);
    this.pending.delete(id);
    return { name: call.name, args: JSON.parse(call.argumentsText) as Record<string, unknown> };
  }
}

Apply the same rule to structured output: accumulate, parse once at a known-complete boundary, validate against your schema, and surface failures as stream errors instead of substituting an empty object.

Retries, Resume, and Their Limits

Streaming retries are weaker than ordinary retries. If a request fails before any bytes arrive, retrying is safe: nothing was shown or executed. Mid-stream, you have partial text and possibly a partial tool call. A transparent retry would restart generation and duplicate content, or require the provider to resume from an exact position — do not assume it can.

SSE defines an event identifier and reconnection, where the client sends the last identifier back. That only helps if your server records events by identifier and can replay or continue from them; adding the field does not give you resume for free.

Retry automatically only before the first event, sparingly, using your existing request layer. For mid-stream failures, stop, preserve the partial output labelled as incomplete, and let the user start a new generation. Key any tool-call side effect to its identifier and record completion, so a retry does not repeat it. Use a separate timeout for time to first event: a long answer can take a while to finish, but a stream that produces nothing should fail soon enough for the user to act.

Honest Loading and Typing States

Streaming UIs fail users less through slowness than through dishonesty: a spinner implying a stage you have not reached, or an answer that looks finished while it is still arriving. Model the UI as states tied to events you actually observe:

• Sending: request created, no response headers yet. Say you are waiting for the model, not generating.
• Waiting for first token: headers arrived, no delta yet. The server has accepted the request.
• Streaming: deltas are arriving. Show partial text with a clear in-progress affordance and a stop control that actually aborts.
• Stopped by user: keep the partial text, label it stopped, and do not append a fabricated ending.
• Incomplete or failed: an error arrived or the connection dropped. Mark the partial text visibly.
• Complete: only an explicit finish or done event moves a message here. Connection closure alone is not proof — a dropped connection also closes the stream.

Do not simulate typing. Render bursts as bursts, smoothing across animation frames if needed, without inventing characters or holding content back for effect. Show phase labels such as retrieval or tool execution only when driven by a real event. For accessibility, announce completion through a live region rather than every token, keep the stop control keyboard-reachable, and do not move focus as content grows.

Proxy Buffering and Deployment Pitfalls

Correct code can stream locally and arrive all at once in production because an intermediary collected the response. Check these first:

• Headers: send Cache-Control: no-cache, no-transform and, for nginx-based proxies, X-Accel-Buffering: no. Disable compression for the streaming route or confirm the compressor flushes per chunk — a compressor waiting to fill its buffer defeats streaming.
• Platform buffering: confirm your platform, serverless runtime, or gateway supports streaming in your configuration. Verify on the deployed endpoint with a command-line HTTP client that prints chunks as they arrive.
• Timeouts: intermediaries impose idle and total limits. A periodic heartbeat — an SSE comment such as : ping followed by a blank line — stops a quiet stream being treated as dead during long tool execution, but does not extend a hard total limit.
• Concurrency: each open stream holds a connection for its full duration. Size for concurrent open streams, not just requests per second, and ensure aborted streams release their slot promptly.

Verdict: What to Build by Default

Default to an HTTP response streamed as server-sent events, read with fetch and response.body.getReader(). You get POST, custom headers, a standard debuggable format, in-band errors, and abort propagation from stop button to provider call — with no socket infrastructure to operate. Choose WebSockets only when the client must send messages on the same channel during generation, and a custom chunked format only when you already own a protocol SSE cannot express.

Five behaviours are non-negotiable: frame events explicitly, accumulate tool-call arguments to a known-complete boundary before parsing or executing, wire cancellation end to end, retry automatically only before the first event, and drive every loading label from an event you observed. Get those right and streaming becomes a truthful view of a model still at work, at the pace the network, server, and reader can each sustain.