> ## Documentation Index
> Fetch the complete documentation index at: https://docs.requesty.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming Responses

> Real-time response streaming for improved user experience and reduced perceived latency

<Info>
  Streaming responses provide immediate feedback to users by delivering content token-by-token as it's generated, dramatically improving perceived performance and user experience.

  <Note>
    **[Get your API key](https://app.requesty.ai/api-keys)** in the Requesty Console.
  </Note>
</Info>

## Overview

Requesty supports streaming responses from all major providers (OpenAI, Anthropic, Google, Mistral) using Server-Sent Events (SSE). Instead of waiting for the complete response, your applications can display content as it's being generated.

## Why Use Streaming?

<CardGroup cols={2}>
  <Card title="Improved User Experience" icon="users">
    Users see responses immediately, reducing perceived wait time by up to 80%
  </Card>

  <Card title="Better Engagement" icon="heart">
    Real-time content delivery keeps users engaged during longer responses
  </Card>

  <Card title="Reduced Timeouts" icon="clock">
    Avoid timeout issues on slow or complex requests
  </Card>

  <Card title="Progressive Display" icon="bars-progress">
    Enable progressive UI updates as content becomes available
  </Card>
</CardGroup>

## Implementation

### Basic Streaming Setup

Enable streaming by setting the `stream` parameter to `true` in your request:

<CodeGroup>
  ```python Python theme={"dark"}
  import openai

  client = openai.OpenAI(
      api_key="your_requesty_api_key",
      base_url="https://router.requesty.ai/v1",
  )

  response = client.chat.completions.create(
      model="openai/gpt-4",
      messages=[{"role": "user", "content": "Write a poem about the stars."}],
      stream=True
  )

  # Process streaming response
  for chunk in response:
      if chunk.choices[0].delta.content is not None:
          content = chunk.choices[0].delta.content
          print(content, end="", flush=True)
  ```

  ```javascript JavaScript theme={"dark"}
  import OpenAI from 'openai';

  const openai = new OpenAI({
    apiKey: 'your_requesty_api_key',
    baseURL: 'https://router.requesty.ai/v1',
  });

  const stream = await openai.chat.completions.create({
    model: 'openai/gpt-4',
    messages: [{ role: 'user', content: 'Write a poem about the stars.' }],
    stream: true,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      process.stdout.write(content);
    }
  }
  ```

  ```curl cURL theme={"dark"}
  curl -X POST "https://router.requesty.ai/v1/chat/completions" \
    -H "Authorization: Bearer your_requesty_api_key" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-4",
      "messages": [{"role": "user", "content": "Write a poem about the stars."}],
      "stream": true
    }' \
    --no-buffer
  ```
</CodeGroup>

### Token Usage and Cost While Streaming

By default, streaming responses do **not** include a `usage` object, only the content deltas. To receive token counts and the Requesty `cost` for the request, opt in with the OpenAI-compatible `stream_options: {"include_usage": true}` parameter. When enabled, an extra chunk is sent right before `data: [DONE]` containing the full `usage` object (with an empty `choices` array).

<CodeGroup>
  ```python Python theme={"dark"}
  response = client.chat.completions.create(
      model="openai/gpt-4o",
      messages=[{"role": "user", "content": "Write a poem about the stars."}],
      stream=True,
      stream_options={"include_usage": True},
  )

  for chunk in response:
      if chunk.choices and chunk.choices[0].delta.content:
          print(chunk.choices[0].delta.content, end="", flush=True)
      # The final chunk has no choices but carries usage + cost
      if chunk.usage:
          print(f"\nTokens: {chunk.usage.total_tokens}  Cost: ${chunk.usage.cost}")
  ```

  ```javascript JavaScript theme={"dark"}
  const stream = await openai.chat.completions.create({
    model: 'openai/gpt-4o',
    messages: [{ role: 'user', content: 'Write a poem about the stars.' }],
    stream: true,
    stream_options: { include_usage: true },
  });

  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
    if (chunk.usage) {
      console.log(`\nTokens: ${chunk.usage.total_tokens}  Cost: $${chunk.usage.cost}`);
    }
  }
  ```

  ```curl cURL theme={"dark"}
  curl -X POST "https://router.requesty.ai/v1/chat/completions" \
    -H "Authorization: Bearer your_requesty_api_key" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-4o",
      "messages": [{"role": "user", "content": "Write a poem about the stars."}],
      "stream": true,
      "stream_options": {"include_usage": true}
    }' \
    --no-buffer
  ```
</CodeGroup>

<Info>
  For non-streaming requests, `usage` (including `cost`) is returned by default, no extra parameter needed. See the [API Overview](/api-reference/overview#non-streaming-response-example).
</Info>

### Advanced Streaming Patterns

<Tabs>
  <Tab title="Collecting Complete Response">
    Accumulate streaming chunks to build the full response:

    ```python theme={"dark"}
    collected_content = []

    for chunk in response:
        if chunk.choices[0].delta.content is not None:
            content = chunk.choices[0].delta.content
            collected_content.append(content)

    full_response = "".join(collected_content)
    print(f"Complete response: {full_response}")
    ```
  </Tab>

  <Tab title="Function Call Streaming">
    Handle streaming function calls and tool usage:

    ```python theme={"dark"}
    for chunk in response:
        delta = chunk.choices[0].delta

        # Handle regular content
        if delta.content:
            print(delta.content, end="", flush=True)

        # Handle function calls
        if hasattr(delta, 'function_call') and delta.function_call:
            fc = delta.function_call
            if fc.name:
                print(f"\nCalling function: {fc.name}")
            if fc.arguments:
                print(f"Arguments: {fc.arguments}")
    ```
  </Tab>

  <Tab title="Error Handling">
    Implement robust error handling for production use:

    ```python theme={"dark"}
    try:
        for chunk in response:
            if chunk.choices[0].delta.content:
                content = chunk.choices[0].delta.content
                print(content, end="", flush=True)
    except Exception as e:
        print(f"Streaming error: {e}")
        # Implement fallback or retry logic
    ```
  </Tab>
</Tabs>

## Streaming Features

### Supported Capabilities

* **Text Generation**: Standard chat completions
* **Function Calling**: Streaming function calls and arguments
* **Tool Usage**: Tool calls with streaming responses
* **Multi-turn Conversations**: Streaming in conversation contexts
* **System Messages**: Full prompt template support
* **Parameters**: Temperature, max\_tokens, and other standard parameters

### Provider Compatibility

All major providers support streaming through Requesty:

* **OpenAI**: GPT-4, GPT-3.5, and all variants
* **Anthropic**: Claude 3.5 Sonnet, Claude 3 Haiku/Opus
* **Google**: Gemini Pro, Gemini Flash
* **Mistral**: All Mistral models
* **Meta**: Llama models

## Best Practices

<AccordionGroup>
  <Accordion title="Optimize for User Experience">
    * Display content immediately as it arrives
    * Use typing indicators or progress bars
    * Handle partial responses gracefully
    * Implement smooth scrolling for long content
  </Accordion>

  <Accordion title="Handle Network Issues">
    * Implement connection retry logic
    * Gracefully handle stream interruptions
    * Provide fallback to non-streaming mode
    * Monitor stream health and performance
  </Accordion>

  <Accordion title="Performance Optimization">
    * Use `flush=True` for immediate output display
    * Batch UI updates for better performance
    * Implement efficient chunk processing
    * Consider client-side buffering strategies
  </Accordion>

  <Accordion title="Production Considerations">
    * Implement proper error boundaries
    * Log streaming metrics and performance
    * Test with various network conditions
    * Plan for graceful degradation
  </Accordion>
</AccordionGroup>

## Common Use Cases

<CardGroup cols={3}>
  <Card title="Chat Applications" icon="message">
    Real-time messaging with immediate response display
  </Card>

  <Card title="Content Generation" icon="pen">
    Progressive article, blog, or document creation
  </Card>

  <Card title="Code Generation" icon="code">
    Live code generation with syntax highlighting
  </Card>

  <Card title="Data Analysis" icon="chart-simple">
    Streaming analysis results and insights
  </Card>

  <Card title="Creative Writing" icon="feather">
    Story, poem, or creative content generation
  </Card>

  <Card title="Technical Documentation" icon="book">
    Progressive documentation and explanation generation
  </Card>
</CardGroup>

## Integration Examples

### React Component

```jsx theme={"dark"}
function StreamingChat() {
  const [content, setContent] = useState('');

  const handleStream = async () => {
    const response = await fetch('/api/chat/stream', {
      method: 'POST',
      body: JSON.stringify({ message: userInput })
    });

    const reader = response.body.getReader();
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const chunk = new TextDecoder().decode(value);
      setContent(prev => prev + chunk);
    }
  };

  return <div>{content}</div>;
}
```

## Troubleshooting

<Warning>
  Always implement proper error handling when using streaming responses, as network interruptions can cause incomplete responses.
</Warning>

### Common Issues

* **Stream Interruption**: Implement retry logic and graceful fallbacks
* **Partial Responses**: Handle incomplete function calls or content
* **Performance**: Optimize chunk processing for large responses
* **Browser Compatibility**: Test streaming across different browsers and devices