> ## Documentation Index
> Fetch the complete documentation index at: https://docs.requesty.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Load Balancing Policies

> Distribute traffic across models with weighted routing

Load Balancing Policies distribute your requests across multiple models based on weights you define. Perfect for A/B testing, gradual rollouts, and resource optimization.

<Frame caption="Traffic is split across models by the weights you set, while the same conversation can stay on one provider.">
  <img src="https://mintcdn.com/requesty/qjPoKXyN196jjWse/images/loadbalancing_routing.png?fit=max&auto=format&n=qjPoKXyN196jjWse&q=85&s=5427aae4adbee381f9274508986e6a9f" alt="Load balancing: the Requesty router distributes incoming requests across multiple providers according to configured weights, with trace_id keeping a single conversation on the same provider." width="1536" height="1024" data-path="images/loadbalancing_routing.png" />
</Frame>

<Note>
  **[Configure load balancing](https://app.requesty.ai/routing-policies)** in the Requesty Console.
</Note>

## How It Works

<Steps>
  <Step title="Assign weights">
    You assign **weights** to each model (e.g., 70%, 20%, 10%).
  </Step>

  <Step title="Requests are routed">
    Each incoming request is **consistently routed** to one model based on the distribution.
  </Step>

  <Step title="Consistency guaranteed">
    Requests with the same `trace_id` or `user_id` always go to the **same model**.
  </Step>
</Steps>

## Benefits

<CardGroup cols={2}>
  <Card title="A/B Testing" icon="flask">
    Compare model performance with real traffic split across different models.
  </Card>

  <Card title="Gradual Rollouts" icon="chart-line">
    Send 10% to a new model, 90% to your stable model. Increase gradually.
  </Card>

  <Card title="Cost Optimization" icon="piggy-bank">
    Route most traffic to cheaper models while keeping premium models available.
  </Card>

  <Card title="Consistent Experiences" icon="user-check">
    Same user always gets the same model, maintaining conversation context.
  </Card>
</CardGroup>

## Creating a Load Balancing Policy

<Steps>
  <Step title="Create the Policy">
    Go to [Routing Policies](https://app.requesty.ai/routing-policies), click **Create Policy**, and select **Load Balancing** as the policy type.

    <img src="https://mintcdn.com/requesty/PoV3BFgV5ivq6GBX/images/load_balancing.png?fit=max&auto=format&n=PoV3BFgV5ivq6GBX&q=85&s=829a1040bffb092fa13c0198be9889b3" alt="Load Balancing Policy" width="1479" height="460" data-path="images/load_balancing.png" />
  </Step>

  <Step title="Configure Weights">
    Set up your distribution. For example:

    | Model                                    | Weight |
    | ---------------------------------------- | ------ |
    | `anthropic/claude-sonnet-4-5`            | 50%    |
    | `bedrock/claude-sonnet-4-5@eu-central-1` | 50%    |

    The total weights must add up to 100% (you can use any numbers, they are normalized).
  </Step>

  <Step title="Use the Policy in Your Code">
    Reference your policy with `policy/your-policy-name`:

    <CodeGroup>
      ```python Python theme={"dark"}
      from openai import OpenAI

      client = OpenAI(
          base_url="https://router.requesty.ai/v1",
          api_key="your-requesty-api-key"
      )

      response = client.chat.completions.create(
          model="policy/sonnet-distribution",
          messages=[{"role": "user", "content": "Hello!"}]
      )
      ```

      ```typescript TypeScript theme={"dark"}
      import OpenAI from 'openai';

      const client = new OpenAI({
        baseURL: 'https://router.requesty.ai/v1',
        apiKey: 'your-requesty-api-key'
      });

      const response = await client.chat.completions.create({
        model: 'policy/sonnet-distribution',
        messages: [{ role: 'user', content: 'Hello!' }]
      });
      ```

      ```bash cURL theme={"dark"}
      curl https://router.requesty.ai/v1/chat/completions \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer your-requesty-api-key" \
        -d '{
          "model": "policy/sonnet-distribution",
          "messages": [{"role": "user", "content": "Hello!"}]
        }'
      ```
    </CodeGroup>
  </Step>
</Steps>

## Consistency Guarantee

Load balancing uses **deterministic hashing** to ensure the same user always gets the same model.

| Scenario           | Behavior                                                      |
| ------------------ | ------------------------------------------------------------- |
| With `trace_id`    | All requests with the same `trace_id` route to the same model |
| Without `trace_id` | Requesty generates a unique `request_id` for each request     |

This means multi-turn conversations stay on the same model, user sessions get consistent behavior, and A/B test groups are stable.

### Maintaining Consistency Across Requests

To keep a user on the same model across multiple requests, pass a `trace_id`:

<CodeGroup>
  ```python Python theme={"dark"}
  response = client.chat.completions.create(
      model="policy/sonnet-distribution",
      messages=[{"role": "user", "content": "Hello!"}],
      extra_body={
          "requesty": {
              "trace_id": "user-12345"
          }
      }
  )
  ```

  ```typescript TypeScript theme={"dark"}
  const response = await client.chat.completions.create({
    model: 'policy/sonnet-distribution',
    messages: [{ role: 'user', content: 'Hello!' }],
    extra_body: {
      requesty: {
        trace_id: 'user-12345'
      }
    }
  });
  ```
</CodeGroup>

<Tip>
  Use your internal user ID as the `trace_id` to ensure each user gets a consistent model experience while still benefiting from A/B testing.
</Tip>

## Load Balancing Between Policies

You can load balance between **entire routing policies**, not just individual models. This is powerful for canary deployments, A/B testing different routing strategies, and gradual migration from one policy to another.

### Example: Policy Rollout

Say you have two fallback policies and want to gradually shift traffic:

| Policy                                | Models                                       | Weight |
| ------------------------------------- | -------------------------------------------- | ------ |
| `policy/production-fallback` (stable) | openai/gpt-5.2 → anthropic/claude-sonnet-4-5 | 80%    |
| `policy/experimental-fallback` (new)  | google/gemini-2.5-pro → openai/gpt-5.2       | 20%    |

Create a load balancing policy called `gradual-rollout` with these weights. As you gain confidence, adjust to 50/50, then 0/100.

<Warning>
  When load balancing between policies, each policy must be compatible with your request parameters. Do not mix embedding policies with chat completion policies.
</Warning>

## Use Cases

<AccordionGroup>
  <Accordion title="A/B Testing New Models">
    Compare GPT-5.2 vs Gemini 2.5 Pro on real traffic:

    | Model                   | Weight |
    | ----------------------- | ------ |
    | `openai/gpt-5.2`        | 50%    |
    | `google/gemini-2.5-pro` | 50%    |

    Track performance in [Analytics](https://app.requesty.ai/analytics) and see which model performs better.
  </Accordion>

  <Accordion title="Gradual Model Rollout">
    Carefully introduce a new model:

    | Model            | Weight | Role           |
    | ---------------- | ------ | -------------- |
    | `openai/gpt-4o`  | 90%    | Stable, proven |
    | `openai/gpt-5.2` | 10%    | New, testing   |

    Increase the weight of `gpt-5.2` as you validate quality.
  </Accordion>

  <Accordion title="Cost-Optimized Distribution">
    Route most traffic to cheaper models, some to premium:

    | Model                | Weight |
    | -------------------- | ------ |
    | `openai/gpt-4o-mini` | 70%    |
    | `openai/gpt-4o`      | 20%    |
    | `openai/gpt-5.2`     | 10%    |
  </Accordion>

  <Accordion title="Multi-Provider Redundancy">
    Distribute across providers for resilience:

    | Model                         | Weight |
    | ----------------------------- | ------ |
    | `openai/gpt-5.2`              | 40%    |
    | `anthropic/claude-sonnet-4-5` | 40%    |
    | `google/gemini-2.5-pro`       | 20%    |
  </Accordion>
</AccordionGroup>

## Key Selection (BYOK)

For each model in your load balancing policy, you can choose:

| Option                    | Description                           |
| ------------------------- | ------------------------------------- |
| **Requesty provided key** | Use Requesty's managed keys (default) |
| **My own key**            | Use your BYOK credentials             |

## Monitoring and Analytics

<Steps>
  <Step title="Open Analytics">
    Go to [Analytics](https://app.requesty.ai/analytics).
  </Step>

  <Step title="Filter by policy">
    Filter by your policy name to see the **actual distribution** of requests across models.
  </Step>

  <Step title="Compare performance">
    Compare **latency**, **cost**, and **success rates** between models. The distribution should match your configured weights (±2% variance is normal).
  </Step>
</Steps>

## FAQ

<AccordionGroup>
  <Accordion title="How does consistent hashing work?">
    Requesty uses the **xxhash algorithm** on your `trace_id` (or `request_id` if no trace\_id) to deterministically select a model. The same ID always produces the same hash, which maps to the same model.
  </Accordion>

  <Accordion title="What happens if I change the weights?">
    Changing weights will **re-distribute** traffic. Some users may switch to different models. If you need stability, avoid changing weights frequently, or use separate policies for stable vs experimental traffic.
  </Accordion>

  <Accordion title="Can I load balance and have fallback?">
    Yes. Create a load balancing policy that points to **fallback policies**. This gives you both load balancing and automatic failover.

    | Policy                      | Weight |
    | --------------------------- | ------ |
    | `policy/openai-fallback`    | 50%    |
    | `policy/anthropic-fallback` | 50%    |
  </Accordion>

  <Accordion title="Do all models need to be compatible?">
    Yes. All models in a load balancing policy should support the same request format and features. Do not mix chat models with embedding models, or models with different context lengths.
  </Accordion>

  <Accordion title="How do I ensure exactly 20% of users see the new model?">
    Use a stable `trace_id` (like user ID). With 100+ unique users, the distribution will converge to your configured weights (e.g., 20%). With small sample sizes, expect ±5% variance.
  </Accordion>
</AccordionGroup>