Streaming and cancellation

Read SSE events, handle partial responses, and cancel generation.

Streaming lets your application display output as it arrives. Set stream: true; the response uses Server-Sent Events instead of one JSON document.

Request a stream

curl --no-buffer --fail-with-body https://api.simplellm.eu/v1/chat/completions \
  -H "Authorization: Bearer $SIMPLELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-flash-next",
    "messages": [{"role": "user", "content": "Explain how an HTTP request is handled."}],
    "max_tokens": 2048,
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

The usage option requests a final usage report where supported. It cannot guarantee a report after an interrupted connection.

Understand the events

This shortened example illustrates event framing rather than a complete provider-specific response:

data: {"choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"content":"An HTTP request"},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"choices":[],"usage":{"prompt_tokens":20,"completion_tokens":40,"total_tokens":60}}

data: [DONE]

• Append text from choices[].delta.content; do not replace the entire response with each delta.
• An event can contain a role, reasoning, tool-call arguments, or usage instead of visible text.
• Usage-only events may have an empty choices array.
• A TCP/HTTP read can contain part of an event or several events; it is not a token boundary.
• If writing an SSE parser, buffer incomplete lines, support CRLF, and terminate an event at a blank line; join multiple data lines before JSON parsing.

Use an existing SDK's stream reader when possible. Do not count events or network chunks as exact tokens.

Python stream with explicit cleanup

Install the openai package and set SIMPLELLM_API_KEY and SIMPLELLM_MODEL as described in the Quickstart.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.simplellm.eu/v1",
    api_key=os.environ["SIMPLELLM_API_KEY"],
    max_retries=0,
    timeout=600.0,
)
stream = client.chat.completions.create(
    model=os.environ["SIMPLELLM_MODEL"],
    messages=[{"role": "user", "content": "Explain request streaming."}],
    max_tokens=2048,
    stream=True,
    stream_options={"include_usage": True},
)
try:
    for chunk in stream:
        if chunk.usage is not None:
            print("\nUsage:", chunk.usage)
        for choice in chunk.choices:
            if choice.delta.content:
                print(choice.delta.content, end="", flush=True)
finally:
    stream.close()

The client timeout here is an application choice, not a SimpleLLM platform timeout guarantee. Interrupting the program exits the loop and closes the stream in the cleanup block.

Cancel the upstream request

When the user presses Stop, close or abort the HTTP request; merely hiding the output in the UI does not cancel generation. If a browser talks to your backend, propagate the browser's disconnect to your backend's upstream request as well.

In the JavaScript SDK, abort the stream's controller when the application cancels:

// Inside the stream loop or your application's Stop handler:
stream.controller.abort();

The complete JavaScript client setup is in client examples.

Interrupted streams

Treat connection closure without the expected completion signals as an incomplete result. Preserve delivered text with an incomplete/cancelled state so the user can distinguish it from a successful answer.

An HTTP 200 only confirms that streaming began. After headers have been sent, a later failure cannot reliably become a new HTTP error status; inspect stream events and connection errors too.

Do not silently replay a request after partial output. Retrying creates a new generation that can duplicate content, tool actions, and charges.