> ## Documentation Index
> Fetch the complete documentation index at: https://docs.requesty.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Load Balancing Policies

> Distribute traffic across models with weighted routing

Load Balancing Policies distribute your requests across multiple models based on weights you define. Perfect for A/B testing, gradual rollouts, and resource optimization.

![Load balancing: the Requesty router distributes incoming requests across multiple providers according to configured weights, with trace\_id keeping a single conversation on the same provider.](https://mintcdn.com/requesty/qjPoKXyN196jjWse/images/loadbalancing_routing.png?fit=max\&auto=format\&n=qjPoKXyN196jjWse\&q=85\&s=5427aae4adbee381f9274508986e6a9f)

Traffic is split across models by the weights you set, while the same conversation can stay on one provider.

> [!NOTE]
>
> **[Configure load balancing](https://app.requesty.ai/routing-policies)** in the Requesty Console. Prefer zero setup? Try a [managed policy](https://docs.requesty.ai/features/managed-policies.md) maintained by Requesty.

## How It Works

**1. Assign weights**

You assign **weights** to each model (e.g., 70%, 20%, 10%).

**2. Requests are routed**

Each incoming request is **consistently routed** to one model based on the distribution.

**3. Consistency guaranteed**

Requests with the same `trace_id` or `user_id` always go to the **same model**.

## Benefits

> [card] A/B Testing
>
> Compare model performance with real traffic split across different models.

> [card] Gradual Rollouts
>
> Send 10% to a new model, 90% to your stable model. Increase gradually.

> [card] Cost Optimization
>
> Route most traffic to cheaper models while keeping premium models available.

> [card] Consistent Experiences
>
> Same user always gets the same model, maintaining conversation context.

## Creating a Load Balancing Policy

**1. Create the Policy**

Go to [Routing Policies](https://app.requesty.ai/routing-policies), click **Create Policy**, and select the **Load balance** strategy card. Name your policy and reference it as `policy/your-policy-name` in your requests.

![Policy editor with strategy cards and Live Performance panel](https://mintcdn.com/requesty/TlHWMQ2xwwK_DKt5/images/routing-policy-editor.png?fit=max\&auto=format\&n=TlHWMQ2xwwK_DKt5\&q=85\&s=fe659370d487574f2dc7463a25d7cb4d)

**2. Configure Weights**

Add models from the catalog below the form, then set your distribution. For example:

| Model | Weight |
| - | - |
| `anthropic/claude-sonnet-5-5` | 50% |
| `bedrock/claude-sonnet-5-5@eu-central-1` | 50% |

The total weights must add up to 100% (you can use any numbers, they are normalized).

**3. Use the Policy in Your Code**

Reference your policy with `policy/your-policy-name`:

```python Python theme={"dark"}
from openai import OpenAI

client = OpenAI(
    base_url="https://router.requesty.ai/v1",
    api_key="your-requesty-api-key"
)

response = client.chat.completions.create(
    model="policy/sonnet-distribution",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

```typescript TypeScript theme={"dark"}
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://router.requesty.ai/v1',
  apiKey: 'your-requesty-api-key'
});

const response = await client.chat.completions.create({
  model: 'policy/sonnet-distribution',
  messages: [{ role: 'user', content: 'Hello!' }]
});
```

```bash cURL theme={"dark"}
curl https://router.requesty.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-requesty-api-key" \
  -d '{
    "model": "policy/sonnet-distribution",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

## Consistency Guarantee

Load balancing uses **deterministic hashing** to ensure the same user always gets the same model.

| Scenario | Behavior |
| - | - |
| With `trace_id` | All requests with the same `trace_id` route to the same model |
| Without `trace_id` | Requesty generates a unique `request_id` for each request |

This means multi-turn conversations stay on the same model, user sessions get consistent behavior, and A/B test groups are stable.

### Maintaining Consistency Across Requests

To keep a user on the same model across multiple requests, pass a `trace_id`:

```python Python theme={"dark"}
response = client.chat.completions.create(
    model="policy/sonnet-distribution",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "requesty": {
            "trace_id": "user-12345"
        }
    }
)
```

```typescript TypeScript theme={"dark"}
const response = await client.chat.completions.create({
  model: 'policy/sonnet-distribution',
  messages: [{ role: 'user', content: 'Hello!' }],
  extra_body: {
    requesty: {
      trace_id: 'user-12345'
    }
  }
});
```

> [!TIP]
>
> Use your internal user ID as the `trace_id` to ensure each user gets a consistent model experience while still benefiting from A/B testing.

## Load Balancing Between Policies

You can load balance between **entire routing policies**, not just individual models. This is powerful for canary deployments, A/B testing different routing strategies, and gradual migration from one policy to another.

### Example: Policy Rollout

Say you have two fallback policies and want to gradually shift traffic:

| Policy | Models | Weight |
| - | - | - |
| `policy/production-fallback` (stable) | openai/gpt-6.1-sol → anthropic/claude-sonnet-5-5 | 80% |
| `policy/experimental-fallback` (new) | google/gemini-3.1-pro-preview → openai/gpt-6.1-sol | 20% |

Create a load balancing policy called `gradual-rollout` with these weights. As you gain confidence, adjust to 50/50, then 0/100.

> [!WARNING]
>
> When load balancing between policies, each policy must be compatible with your request parameters. Do not mix embedding policies with chat completion policies.

## Use Cases

> [accordion] A/B Testing New Models
>
> Compare GPT-5.2 vs Gemini 2.5 Pro on real traffic:
>
> | Model | Weight |
> | - | - |
> | `openai/gpt-6.1-sol` | 50% |
> | `google/gemini-3.1-pro-preview` | 50% |
>
> Track performance in [Analytics](https://app.requesty.ai/analytics) and see which model performs better.

> [accordion] Gradual Model Rollout
>
> Carefully introduce a new model:
>
> | Model | Weight | Role |
> | - | - | - |
> | `openai/gpt-6-sol` | 90% | Stable, proven |
> | `openai/gpt-6.1-sol` | 10% | New, testing |
>
> Increase the weight of `gpt-6.1-sol` as you validate quality.

> [accordion] Cost-Optimized Distribution
>
> Route most traffic to cheaper models, some to premium:
>
> | Model | Weight |
> | - | - |
> | `openai/gpt-6-luna` | 70% |
> | `openai/gpt-6-sol` | 20% |
> | `openai/gpt-6.1-sol` | 10% |

> [accordion] Multi-Provider Redundancy
>
> Distribute across providers for resilience:
>
> | Model | Weight |
> | - | - |
> | `openai/gpt-6.1-sol` | 40% |
> | `anthropic/claude-sonnet-5-5` | 40% |
> | `google/gemini-3.1-pro-preview` | 20% |

## Key Selection (BYOK)

For each model in your load balancing policy, you can choose:

| Option | Description |
| - | - |
| **Requesty provided key** | Use Requesty's managed keys (default) |
| **My own key** | Use your BYOK credentials |

## Monitoring and Analytics

Every policy page includes a **Live Performance** panel showing real traffic across the models in the policy. Switch between **Latency**, **Success**, and **Speed** over the last 24 hours or 7 days to see how the distribution plays out in practice.

For deeper analysis:

**1. Open Analytics**

Go to [Analytics](https://app.requesty.ai/analytics).

**2. Filter by policy**

Filter by your policy name to see the **actual distribution** of requests across models.

**3. Compare performance**

Compare **latency**, **cost**, and **success rates** between models. The distribution should match your configured weights (±2% variance is normal).

## FAQ

> [accordion] How does consistent hashing work?
>
> Requesty uses the **xxhash algorithm** on your `trace_id` (or `request_id` if no trace\_id) to deterministically select a model. The same ID always produces the same hash, which maps to the same model.

> [accordion] What happens if I change the weights?
>
> Changing weights will **re-distribute** traffic. Some users may switch to different models. If you need stability, avoid changing weights frequently, or use separate policies for stable vs experimental traffic.

> [accordion] Can I load balance and have fallback?
>
> Yes. Create a load balancing policy that points to **fallback policies**. This gives you both load balancing and automatic failover.
>
> | Policy | Weight |
> | - | - |
> | `policy/openai-fallback` | 50% |
> | `policy/anthropic-fallback` | 50% |

> [accordion] Do all models need to be compatible?
>
> Yes. All models in a load balancing policy should support the same request format and features. Do not mix chat models with embedding models, or models with different context lengths.

> [accordion] How do I ensure exactly 20% of users see the new model?
>
> Use a stable `trace_id` (like user ID). With 100+ unique users, the distribution will converge to your configured weights (e.g., 20%). With small sample sizes, expect ±5% variance.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.