Python

Server SDKRuns on your backend (Calling Key)

Published to PyPIpython -m pip install "abto[openai]"

The Python SDK automatically carries the call-time user and feature context on gateway requests. Tokens, cost, latency, and the request identifier are recorded by the gateway.

import os

from abto import init_abto

abto = init_abto(
    api_key=os.environ["ABTO_CALLING_KEY"],
    gateway_base_url="https://gateway.abto.app/v1",
    provider_keys={
        "openai": os.environ["OPENAI_API_KEY"],
        # Pass candidate keys together when the project routes across providers.
        "anthropic": os.environ["ANTHROPIC_API_KEY"],
        "gemini": os.environ["GEMINI_API_KEY"],
        "deepseek": os.environ["DEEPSEEK_API_KEY"],
        "kimi": os.environ["KIMI_API_KEY"],
    },
)

openai = abto.openai()

with abto.with_context(
    device_id="device-abc",
    feature_id="review.summary",
):
    response = openai.chat.completions.with_raw_response.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": "Summarize these 12 product reviews in three lines."}],
    )
    request_id = response.headers.get("x-abto-request-id")
    completion = response.parse()

• completion: the usual OpenAI response body
• raw response: where you read the x-abto-request-id the Gateway issued
• gateway_base_url: required. Omit it and the SDK reads ABTO_GATEWAY_BASE_URL; with neither it raises ValueError

Connect LangChain

Pass ABTO’s connection options to your existing ChatOpenAI model. Keep your chains, retrievers, prompts, callbacks, and retry settings.

pip install "abto[openai]" langchain-openai

This example connects both sync and async calls. Set your Calling Key and OpenAI API key in server environment variables.

import asyncio
import os

from abto import init_abto
from langchain_openai import ChatOpenAI

abto = init_abto(
    api_key=os.environ["ABTO_CALLING_KEY"],
    gateway_base_url="https://gateway.abto.app/v1",
    provider_keys={"openai": os.environ["OPENAI_API_KEY"]},
)

sync_options = abto.openai_options()
async_options = abto.async_openai_options()
model = ChatOpenAI(
    model="gpt-4.1-mini",
    use_responses_api=False,
    **sync_options,
    http_async_client=async_options["http_client"],
)

async def main():
    try:
        with abto.with_context(feature_id="support.reply"):
            reply = await model.ainvoke("How can I check my delivery status?")
            print(reply.content)
    finally:
        sync_options["http_client"].close()
        await async_options["http_client"].aclose()

asyncio.run(main())

In synchronous code, call model.invoke(...) on the same model. In a server, create and reuse the model and connection options, then close the HTTP clients at shutdown. After a call, find support.reply in the Dashboard request list to check the response, cost, and latency.

Reuse an existing HTTP client

Wrap the transport where you create your existing client, then pass the client to ABTO. Your custom send() method, request/response hooks, connection pools, and cookies stay in use.

import httpx
from abto import wrap_httpx_transport

# If you already have a transport, wrap that object instead of HTTPTransport().
existing_sync_client = httpx.Client(
    transport=wrap_httpx_transport(httpx.HTTPTransport()),
    trust_env=False,
)
sync_options = abto.openai_options(http_client=existing_sync_client)

existing_async_client = httpx.AsyncClient(
    transport=wrap_httpx_transport(httpx.AsyncHTTPTransport()),
    trust_env=False,
)
async_options = abto.async_openai_options(http_client=existing_async_client)

Keep your existing timeout, TLS, and proxy settings. This example disables automatic proxy selection from environment variables. If you use a proxy, configure it explicitly on the transport. Wrap each transport in mounts as well. ABTO does not replace the internals of an already-created client; make this change where you construct the client. An unwrapped route receives no ABTO credentials, and ABTO reports a configuration error after receiving its response.

Closing the returned ABTO clients does not close the clients you supplied. Keep their existing shutdown handling. ABTO owns Gateway and direct-fallback authentication and disables automatic redirects to keep keys on the configured destinations. After customer hooks run, ABTO checks the destination and Host header, then adds keys only to a separate wire request. Your request/response hooks and send() do not receive real ABTO credentials added by the SDK. Errors from your hooks or observation code in send() propagate without triggering ABTO direct fallback. Your existing OpenAI and LangChain retry settings still apply. Place manually configured logging transports outside the ABTO wrapper. The inner transport that performs network I/O necessarily receives authentication keys.

You can also pass openai_options() to the official OpenAI client and async_openai_options() to AsyncOpenAI. The returned api_key is a placeholder that keeps the real key out of framework trace serialization. Credentials are injected when the request is sent.

Keep request context and tracing

Invoke each user’s chain inside with abto.with_context(feature_id=..., device_id=...). Python contextvars keeps concurrent async requests separate. For background jobs, also store and forward the device ID received from the browser with the job data.

Keep Langfuse or LangSmith callbacks, tags, and metadata in your existing invocation configuration. Tracing tools record the input your app sends. If you change the model, prompt, or parameters in the Dashboard, check the ABTO request record for what actually ran. The Gateway may remove tracing headers before forwarding to the provider, so a single trace extending through the provider is not guaranteed.

This integration is for OpenAI Chat Completions. Do not use this example to convert calls that use streaming or the Responses API. Keep TXT loaders and vector retrievers in your existing chain, and keep embeddings on their existing provider connection. To continue calls directly during Gateway outages, also configure direct fallback below.

Gateway headers sent on every request

SDK input	Gateway header	Meaning
api_key	Authorization: Bearer …	ABTO Calling Key
provider_keys["openai"]	x-abto-key-openai	Your own OpenAI provider key
provider_keys["gemini"]	x-abto-key-gemini	Your own Gemini provider key
provider_keys["anthropic"]	x-abto-key-anthropic	Your own Anthropic provider key
provider_keys["deepseek"]	x-abto-key-deepseek	Your own DeepSeek provider key
provider_keys["kimi"]	x-abto-key-kimi	Your own Kimi provider key
feature_id	x-abto-feature-id	Feature ID
node_id	x-abto-node-id	Node inside the feature (optional)
device_id	x-abto-device-id	End-user device identifier (optional)
trace_id	x-abto-trace-id	Run ID (created when omitted)

How provider_keys behaves:

• Not a setting that registers keys with the Gateway. It is the input that attaches server-held credentials as per-request headers.
• Passed with request scope only and never stored in call records.
• If the routed provider has no key, the Gateway rejects that request.
• Pass a callable instead of a string to re-evaluate it on every request. for key rotation and per-provider credential resolvers.

How caller-supplied headers are handled:

• Do not override x-abto-key-* through extra_headers on a client built by abto.openai().
• The Python SDK strips the caller’s Authorization, x-abto-key-*, x-abto-feature-id, x-abto-node-id, x-abto-device-id, and x-abto-trace-id, then rebuilds them from the trusted init_abto settings and context.
• Using the official OpenAI SDK directly instead? Put the same headers in each request’s extra_headers. see the direct OpenAI SDK example.

Context fields

The fields mean the same as in Node.js JavaScript.

• feature_id: feature ID (e.g. review.summary), sent as x-abto-feature-id
• node_id: the point inside the feature that calls the LLM (e.g. review.draft). Without it, the feature ID becomes the node name. Sent as x-abto-node-id
  • Example: with abto.with_context(feature_id="review.summary", node_id="review.draft"):
  • When one feature calls the LLM more than once, give each call site its own value to choose a separate agent config per node in Routing
  • If an inner with_context sets a different feature_id without a node_id, it does not inherit the outer node_id
• device_id: sent as x-abto-device-id. Take the browser-generated device_id from your backend request and pass it through so product behavior joins.
• trace_id: the run ID that groups the calls one agent or workflow run makes. Sent as x-abto-trace-id
  • Without it, the outermost with_context creates a new run ID, and inner with_context blocks inherit it
  • Wrap one whole run in a single with_context so every call inside joins the same run. Wrapping each call separately splits the run per call
  • A call made outside any with_context gets its own new run ID
  • When you pass a context directly, as in get_headers(AbtoContext(...)), and leave trace_id empty, the enclosing with_context value is used
  • Set it yourself only to continue a run another service passed to you, in the outermost with_context
  • Accepts up to 200 printable ASCII characters (letters, digits, spaces, symbols). The Gateway ignores an empty or non-conforming value without an error
  • The request’s traceparent is not used as the run ID

Rules for handling identifiers:

• Validate client-supplied device_id and feature_id with your application’s existing request schema.
• Never accept the Calling Key or provider keys from client requests. Read them only from server environment variables or a server-only credential resolver.

Joining responses to browser and mobile behavior:

• Read x-abto-request-id with with_raw_response as shown above.
• Include the same header in your backend response to join the Browser or Mobile SDK LLM trace.

Preserve existing traces

ABTO keeps the traceparent and tracestate headers supplied by your tracing tool, including during direct fallback. It adds feature and device information separately without replacing your trace ID.

This is supported from 1.1.2. Keep your tracing tool’s request header setup. ABTO does not configure tracing for you or connect traces through to the provider.

The run ID trace_id is sent only as x-abto-trace-id, never turned into a traceparent.

OpenAI requests the Gateway supports

• The data path is currently OpenAI Chat Completions.
• Inline base64 images and PDFs in user messages are supported.
• Function tool calling is supported. When you send tool results back, echo each message.tool_calls[].id as tool_call_id (not the top-level response id).
• Unsupported fields such as streaming, remote image URLs, and audio are rejected with 400 rather than silently ignored.
• See Gateway Chat Completions for the exact field list.

Retries

openai = abto.openai(max_retries=2)

• The official OpenAI SDK’s max_retries applies as written. ABTO never overwrites it.
• Leave it unset and the official OpenAI SDK default applies.
• The Calling SDK adds no retry count of its own.
• Direct fallback exposes no count either, only the resend switch fallback.on_timeout, which is off by default.
• Every other official OpenAI option is forwarded unchanged.
• In abto.openai(), api_key, base_url, and http_client are owned by ABTO for trusted routing. passing them raises ValueError rather than being silently ignored.

OpenAI direct fallback on Gateway failures

This is the escape hatch back to the endpoint this application used before ABTO. Name that destination in fallback.base_url; there is no default. If you took an API key straight from OpenAI and used the official SDK, that address is https://api.openai.com/v1.

from abto import OpenAIDirectFallbackOptions, init_abto

abto = init_abto(
    api_key=os.environ["ABTO_CALLING_KEY"],
    gateway_base_url="https://gateway.abto.app/v1",
    provider_keys={"openai": os.environ["OPENAI_API_KEY"]},
    fallback=OpenAIDirectFallbackOptions(
        # The address this code called before ABTO was put in front of it.
        base_url="https://api.openai.com/v1",
        timeout_seconds=30,
        on_timeout=False,
    ),
)

Without it there is no fallback during an ABTO outage

base_url is where fallback sends the request. Leave it unset and fallback stays off, so a Gateway failure has nowhere to fall back to and that request fails outright. Set it if you want your existing path to keep serving traffic after adopting ABTO.

The original Chat Completions body and model go straight to that address; the Gateway’s provider/model policy is not reproduced.

Sends the current request directly

• The request never reached ABTO
• It reached ABTO but was rejected before the provider was called

In other words, it switches only when the provider certainly did not run, so it never creates a double execution or a double charge.

Does not fall back the current request

• Timeouts and disconnects with ambiguous delivery. the Gateway may already have run the provider
• Provider and transport errors; deterministic 4xx and 429; anything after streaming has started
• The direct circuit stays closed in these cases, and if the official OpenAI SDK retries it calls the Gateway again

Settings

• base_url: required. The OpenAI-compatible endpoint used before ABTO. It must accept the OpenAI request path and Authorization: Bearer.
  • Naming this destination is how fallback is turned on. There is no separate enable flag.
  • init_abto raises ValueError for fallback=True with no destination, and for a destination with no OpenAI key.
• timeout_seconds: per-stage inactivity cap for Gateway connection pool wait, connect, write, and response header read. The body after headers and direct requests keep the abto.openai(timeout=...) value.
• on_timeout=True: also resends the timed-out request. An explicit choice that accepts duplicate execution and billing risk.
• fallback=False: turns the whole feature off.

Boundaries

• Per SDK attempt the transport makes the Gateway judgment and the direct send exactly once each.
• Direct calls bypass Gateway policy, ABTO telemetry, and request_id, and must use a model that endpoint supports.
• Only headers OpenAI needs are forwarded; cookies, proxy credentials, and custom Gateway headers are stripped.
• After switching to direct, OpenAI responses and errors are returned to the official OpenAI SDK unchanged, and that SDK decides whether to retry.
• Native direct fallback for Anthropic and Gemini is out of scope today.