Requesty Router

The Requesty Router provides a universal interface with multiple LLM providers. It includes many additional benefits out-of-the-box such as logging, analytics, security and function calling.

Requesty Router acts as a universal interface to multiple LLM providers, similar to to traditional LLM routers, but comes with a lot of additional value:

• Analytics & Logging: Every request and response is tracked, and insights can be sent to a dedicated endpoint for usage analytics, latency measurements, and performance tuning.

• Auto-Tagging: Automatically tags requests with metadata (like function calls used, latencies, topics, tone of voice and much more.) to help with observability and future optimization.

• Function Calling & Tool Use: Similar to OpenAI’s function calling and OpenRouter’s tool interface, our router supports tool invocation to augment LLM capabilities.

All of this is done through a simple, OpenAI-compatible API structure. If you’re already using the OpenAI Python SDK or curl, you can integrate our router by just changing the base_url and providing our API key.

Getting Started

Prerequisites

1. API Key: You need to login at https://app.requesty.ai/sign-up and setup an API-key at https://app.requesty.ai/insight-api.

2. Once you have your API-key you will need to change the base_url of your Openai Client and change it to https://router.requesty.ai/v1

3. This endpoint adheres to a request structure similar to the OpenAI Chat Completion API (link, but with extended features.

Example: Simple Python Client Setup

You can use the standard OpenAI Python client by simply changing openai.api_base and using our API key header:

import os
import openai
from dotenv import load_dotenv

# Load API key from environment variables
load_dotenv()
ROUTER_API_KEY = os.getenv("ROUTER_API_KEY")

if ROUTER_API_KEY is None:
    raise ValueError("ROUTER_API_KEY not found. Please check your .env file.")

try:
    # Initialize OpenAI client
    client = openai.OpenAI(
        api_key=ROUTER_API_KEY,
        base_url="https://router.requesty.ai/v1",
        default_headers={"Authorization": f"Bearer {ROUTER_API_KEY}"}
    )

    # Example request
    response = client.chat.completions.create(
        model="openai/gpt-4o",
        messages=[{"role": "user", "content": "Hello, who are you?"}]
    )

    # Check if the response is successful
    if not response.choices:
        raise Exception("No response choices found.")

    # Print the result
    print(response.choices[0].message.content)

except openai.OpenAIError as e:
    print(f"OpenAI API error: {e}")

except Exception as e:
    print(f"An unexpected error occurred: {e}")

This makes a request to the local router which proxies it to the appropriate model provider, returning a completion response in OpenAI format. Additionally you can add additional information to the headers you can specify (like HTTP-Referer and X-Title) which can help with analytics and app discoverability.

Request Structure

Your request body to /v1/chat/completions closely follows the OpenAI Chat Completion schema:

• Required Fields:

  • messages: An array of message objects with role and content. Roles can be user, assistant, system, or tool.

  • model: The model name. If omitted, defaults to the user’s or payer’s default model. Here is a full list of the supported models.

• Optional Fields:

  • prompt: Alternative to messages for some providers.

  • stream: A boolean to enable Server-Sent Events (SSE) streaming responses.

  • max_tokens, temperature, top_p, etc.: Standard language model parameters.

  • tools / functions: Allows function calling with a schema defined. See OpenAI's function calling documentation for the structure of these requests.

  • tool_choice: Specifies how tool calling should be handled.

  • response_format: For structured responses (some models only).

• Security Level (optional):

  • You can include a custom header X-Security-Level to enforce content security validation. Levels range from none, basic, advanced, enterprise, etc. This ensures that your prompts and responses adhere to domain-specific compliance and security checks. More information about security at the end of this document

Example Request Body

{
  "model": "openai/gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "max_tokens": 200,
  "temperature": 0.7,
  "stream": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string", "description": "City and state"},
            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
          },
          "required": ["location"]
        }
      }
    }
  ]
}

Here, we also provide a tool (get_current_weather) that the model can call if it decides the user request involves weather data.

Some request fields require a different function, for example if you use response_format you'll need to update the request to client.beta.chat.completions.parse and you may want to use the Pydantic or Zod format for your structure.

Response Structure

The response is normalized to an OpenAI-style ChatCompletion object:

• Streaming: If stream: true, responses arrive incrementally as SSE events with data: lines.

• Function Calls (Tool Calls): If the model decides to call a tool, it will return a function_call in the assistant message. You then execute the tool, append the tool’s result as a role: "tool" message, and send a follow-up request. The LLM will then integrate the tool output into its final answer.

Non-Streaming (full) Response Example:

{
  "id": "chatcmpl-xyz123",
  "object": "chat.completion",
  "created": 1687623702,
  "model": "openai/gpt-4o",
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 50,
    "total_tokens": 60
  },
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ]
}

Function Call Example:

If the model decides it needs the weather tool:

{
  "id": "chatcmpl-abc456",
  "object": "chat.completion",
  "created": 1687623800,
  "model": "openai/gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "function_call": {
          "name": "get_current_weather",
          "arguments": "{ "location": "Boston, MA"}"
        }
      },
      "finish_reason": "function_call"
    }
  ]
}

You would then call the get_current_weather function externally, get the result, and send it back as:

{
  "model": "openai/gpt-4o",
  "messages": [
    {"role": "user", "content": "What is the weather in Boston?"},
    {
      "role": "assistant",
      "content": null,
      "function_call": {
        "name": "get_current_weather",
        "arguments": "{ "location": "Boston, MA" }"
      }
    },
    {
      "role": "tool",
      "name": "get_current_weather",
      "content": "{"temperature": "22", "unit": "celsius", "description": "Sunny"}"
    }
  ]
}

The next completion will return a final answer integrating the tool’s response.

Streaming Responses

The router supports streaming responses from all providers (OpenAI, Anthropic, Mistral) using Server-Sent Events (SSE). Streaming allows you to receive and process the response token by token instead of waiting for the complete response.

How to Use Streaming

1. Enable streaming by setting stream=True in your request

2. The response will be a stream of chunks that you need to iterate over

3. Each chunk contains a delta of the response in the same format as the OpenAI API

Python Example with Streaming:

import openai
client = openai.OpenAI(
    api_key=ROUTER_API_KEY,
    base_url="https://router.requesty.ai/v1",
    default_headers={"Authorization": f"Bearer {ROUTER_API_KEY}"}
)
response = client.chat.completions.create(
    model="openai/gpt-4",
    messages=[{"role": "user", "content": "Write a poem about the stars."}],
    stream=True
)
# Iterate over the stream and handle chunks
for chunk in response:
    # Access content from the chunk (if present)
    if chunk.choices[0].delta.content is not None:
        content = chunk.choices[0].delta.content
        print(content, end="", flush=True)  # Print content as it arrives
    # Handle function calls in streaming (if present)
    if hasattr(chunk.choices[0].delta, 'function_call'):
        fc = chunk.choices[0].delta.function_call
        if hasattr(fc, 'name') and fc.name:
            print(f"\nFunction Call: {fc.name}")
        if hasattr(fc, 'arguments') and fc.arguments:
            print(f"Arguments: {fc.arguments}")

Important Notes

1. Content Access:

   • Always check if delta.content is not None before using it

   • Content comes in small chunks that you may want to collect into a full response

2. Function Calls:

   • Function calls are also streamed and come through the delta.function_call property

   • Check for both name and arguments as they might come in separate chunks

3. Error Handling:

   • Wrap streaming code in try/except to handle potential connection issues

   • The stream might end early if there are errors

4. Best Practices:

   • Use flush=True when printing to see output immediately

   • Consider collecting chunks if you need the complete response

   • For production, implement proper error handling and retry logic

Example: Collecting Complete Response

collected_messages = []
for chunk in response:
    if chunk.choices[0].delta.content is not None:
        content = chunk.choices[0].delta.content
        collected_messages.append(content)

full_response = "".join(collected_messages)

Supported Features in Streaming

• Text completion streaming

• Function calling streaming

• Tool calls streaming

• System messages

• Temperature and other parameters

Analytics & Logging

Every request and response is logged. Additional metadata like latencies you can add additional information such as user_id or location or any metadata sending it in the meta field , enabling you to:

• Measure end-to-end latency.

• Track usage and cost.

• Inspect tool calls and security violations.

• Optimize prompt design based on user behavior.

No additional configuration is needed if you’re okay with default behavior.

Router Summary

Our router provides a familiar, OpenAI-like interface enriched with analytics, safety, and advanced routing. With minimal changes (just switch base_url and provide the right API keys and headers), you can leverage multiple LLM providers, function calling, logging, auto-tagging, and security checks.

Key Points:

• Drop-in replacement for OpenAI endpoints.

• Integrate tool calls easily.

• Receive detailed analytics for every request.

• Use streaming SSE responses for real-time token generation.

• Enjoy multimodal and structured response format support.

Try it out, explore advanced settings, and build secure, observable, and powerful LLM-driven applications with ease!