Post 23

Streaming LLM Responses in the App Router

Token-by-token output, tool calls, cancellation, and resumable streams — the plumbing behind a chat UI that feels instant. Built on Web Streams, Server Actions, and the Vercel AI SDK, with the raw version underneath so you know what it is doing.

Sep 3, 2026/13 min readAI

Share Y

▮~/journal/streaming-llm-responses-in-the-app-router.md

Published: Sep 3, 2026
Reading: 13 min read
Category: AI
Length: ~923 words

5 sections

A chat interface that waits for the full model response before showing anything feels broken, even when it is fast. The fix is streaming: render tokens as they arrive. The App Router is well suited to this because the whole request/response pipeline speaks Web Streams natively — but there are four things that separate a demo from something you would ship.

Streaming the tokens without buffering them somewhere along the way.
Handling tool calls — the model asks to run a function, you run it, you feed the result back.
Cancellation — the user hits stop, or navigates away, and you stop paying for tokens.
Resumability — a dropped connection does not lose the half-written answer.

The raw version: a Route Handler that streams

Before reaching for a library, it is worth seeing the primitive. A Route Handler can return a ReadableStream, and the platform flushes each chunk to the client as it is enqueued.

▮app/api/chat/route.ts — no SDK

1export const runtime = "edge";2 3export async function POST(req: Request) {4  const { messages } = await req.json();5 6  const upstream = await fetch("https://api.openai.com/v1/chat/completions", {7    method: "POST",8    headers: {9      "content-type": "application/json",10      authorization: `Bearer ${process.env.OPENAI_API_KEY}`,11    },12    body: JSON.stringify({ model: "gpt-4o-mini", messages, stream: true }),13  });14 15  const stream = new ReadableStream({16    async start(controller) {17      const reader = upstream.body!.getReader();18      const decoder = new TextDecoder();19      const encoder = new TextEncoder();20 21      while (true) {22        const { done, value } = await reader.read();23        if (done) break;24 25        // OpenAI sends SSE lines: `data: {json}\n\n`26        for (const line of decoder.decode(value).split("\n")) {27          if (!line.startsWith("data: ")) continue;28          const payload = line.slice(6);29          if (payload === "[DONE]") { controller.close(); return; }30          const token = JSON.parse(payload).choices[0]?.delta?.content;31          if (token) controller.enqueue(encoder.encode(token));32        }33      }34      controller.close();35    },36  });37 38  return new Response(stream, {39    headers: { "content-type": "text/plain; charset=utf-8" },40  });41}

Watch out

The one non-obvious failure: some proxies and hosting layers buffer responses without a Content-Type of text/event-stream or an explicit Transfer-Encoding: chunked. If your tokens arrive all at once in production but stream fine locally, that is the cause — set the SSE content type and disable buffering at the CDN.

Reading the stream on the client

▮components/chat.tsx

tsx

1"use client";2import { useState, useRef } from "react";3 4export function Chat() {5  const [text, setText] = useState("");6  const [streaming, setStreaming] = useState(false);7  const abortRef = useRef<AbortController | null>(null);8 9  async function send(messages: Message[]) {10    abortRef.current = new AbortController();11    setStreaming(true);12    setText("");13 14    const res = await fetch("/api/chat", {15      method: "POST",16      body: JSON.stringify({ messages }),17      signal: abortRef.current.signal,18    });19 20    const reader = res.body!.getReader();21    const decoder = new TextDecoder();22    while (true) {23      const { done, value } = await reader.read();24      if (done) break;25      setText((prev) => prev + decoder.decode(value, { stream: true }));26    }27    setStreaming(false);28  }29 30  return (31    <>32      <p>{text}</p>33      {streaming && (34        <button onClick={() => abortRef.current?.abort()}>Stop</button>35      )}36    </>37  );38}

abort() propagates all the way to the upstream fetch — the model API sees the connection close and stops generating. That is the whole cancellation story, and it is free.

The SDK version, and why you want it

The raw version breaks down the moment you add tool calls, structured output, or multiple providers. The Vercel AI SDK handles the protocol so you write the intent.

▮app/api/chat/route.ts — with the AI SDK

1import { openai } from "@ai-sdk/openai";2import { streamText, tool } from "ai";3import { z } from "zod";4 5export async function POST(req: Request) {6  const { messages } = await req.json();7 8  const result = streamText({9    model: openai("gpt-4o"),10    messages,11    tools: {12      getWeather: tool({13        description: "Current weather for a city",14        parameters: z.object({ city: z.string() }),15        execute: async ({ city }) => {16          const data = await fetchWeather(city);17          return { tempC: data.temp, condition: data.summary };18        },19      }),20    },21    maxSteps: 3, // model → tool → model, up to 3 hops22  });23 24  return result.toDataStreamResponse();25}

▮components/chat.tsx — with useChat

tsx

1"use client";2import { useChat } from "@ai-sdk/react";3 4export function Chat() {5  const { messages, input, handleInputChange, handleSubmit, stop, status } =6    useChat();7 8  return (9    <div>10      {messages.map((m) => (11        <div key={m.id} data-role={m.role}>12          {m.parts.map((part, i) =>13            part.type === "text" ? <span key={i}>{part.text}</span> :14            part.type === "tool-invocation" ? (15              <ToolCard key={i} invocation={part.toolInvocation} />16            ) : null,17          )}18        </div>19      ))}20 21      <form onSubmit={handleSubmit}>22        <input value={input} onChange={handleInputChange} />23        {status === "streaming"24          ? <button type="button" onClick={stop}>Stop</button>25          : <button type="submit">Send</button>}26      </form>27    </div>28  );29}

Tip

Render the tool-call parts, do not hide them. Showing “Checking the weather in Berlin…” while the tool runs makes the multi-second gap feel intentional instead of frozen.

Persisting and resuming

Streaming and durability pull in opposite directions: the response only exists as it flows past. To make it resumable, tee the stream — one branch to the client, one to your database — and write the final message when the stream closes.

▮onFinish — write the completed message

1const result = streamText({2  model: openai("gpt-4o"),3  messages,4  async onFinish({ text, usage, finishReason }) {5    await db.message.create({6      data: {7        chatId,8        role: "assistant",9        content: text,10        tokensIn: usage.promptTokens,11        tokensOut: usage.completionTokens,12        finishReason,13      },14    });15  },16});

For true mid-stream resume — the user refreshes while the answer is still generating — you need the stream to have an id the client can reconnect to, and a store (Redis, a durable object) holding the partial text. The AI SDK ships an experimental resumable-stream helper for exactly this; most products do not need it, and “regenerate” is a fine fallback.

Cost and abuse controls belong on the server

Rate-limit per user and per IP in the Route Handler, before you call the model.
Cap maxTokens and message history length — an unbounded context window is an unbounded bill.
Log usage from onFinish so you can see spend per user and per feature.
Never put the API key anywhere near the client. The stream goes browser → your server → model, never browser → model.

Found this useful? Pass it on.

X LinkedIn YHacker News

All posts

Want a custom write-up for your team? Get in touch.