Five Features That Break After a base_url Switch

"Just swap the base_url" holds about 80% of the time. The other 20% doesn't fail loudly — it degrades quietly. HTTP 200, valid response shape, clean logs, but half your tool calls vanished, the stream has no usage block, and the text in the screenshot can't be read. These are the worst bugs to chase because there's no error line to grep.

Here are the five capabilities that get thinned out most often after a migration — the symptom, why it happens, and a ten-line probe for each. Running all five takes about ten minutes and costs under a cent.

The failure matrix first#

Two more are easy to miss: Anthropic's /v1/messages and OpenAI's /v1/responses. Neither is a "compatible" target — they're vendor protocols, and support depends on whether the gateway implemented them. The protocol-level differences are covered in the OpenAI-compatible protocol explainer; this post is about testing.

1. Function calling: serialized calls and broken JSON#

A compat layer usually translates your tool definitions into the target model's native format. Two things fall out during that translation: parallel_tool_calls and strict.

The symptoms are specific:

• You ask for weather and time in one turn and get a single tool_calls entry. The other intent got written into content as prose, and your dispatcher never sees it
• arguments is a string, but occasionally wrapped in ``json fences, carries a trailing comma, or gets truncated by max_tokens — json.loads raises JSONDecodeError`
• A schema declared "strict": true still comes back as {"city": null}, violating required

The probe below hinges on the word "both" — force parallelism:

import json, os, requests

BASE = os.environ.get("ZAIMOD_BASE", "https://api.zaimod.com/v1")
KEY  = os.environ["ZAIMOD_API_KEY"]

tools = [
  {"type": "function", "function": {
    "name": "get_weather", "description": "Weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}},
  {"type": "function", "function": {
    "name": "get_time", "description": "Current time in a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}},
]

r = requests.post(f"{BASE}/chat/completions",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"model": "gpt-4o-mini", "temperature": 0,
          "messages": [{"role": "user", "content": "Get both the weather and the current time for Hangzhou. You must call both tools."}],
          "tools": tools, "tool_choice": "auto"},
    timeout=60).json()

calls = r["choices"][0]["message"].get("tool_calls") or []
print("tool_calls =", len(calls), "(expected 2)")
for c in calls:
    try:
        json.loads(c["function"]["arguments"])
    except json.JSONDecodeError:
        print("invalid JSON:", c["function"]["arguments"][:80])

tool_calls = 1 doesn't make the endpoint unusable; it means your dispatcher has to loop over every call and also scan content for intents that never became calls. Three layers of defence work well in practice: repair the JSON (strip fences, close quotes), if repair fails feed the raw text back and ask for a rewrite, and if that fails fall back to a plain-text answer. Treat strict as advisory and re-validate with pydantic on your side.

2. Streaming: the missing usage block#

The official API ignores usage in streaming mode unless you pass stream_options: {"include_usage": true}, which adds a final chunk with token counts. Compat layers frequently eat that parameter. The consequence is subtle and expensive: your billing module records zero forever, and you find out at month end.

The more painful variant is proxy timeout. A reverse proxy sitting at a 60-second idle limit will cut a long generation mid-flight; the frontend shows "the answer just stopped", and the backend sees nothing wrong because the TCP connection closed cleanly.

One command surfaces both:

curl -sN https://api.zaimod.com/v1/chat/completions \
  -H "Authorization: Bearer $ZAIMOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,
       "stream_options":{"include_usage":true},
       "messages":[{"role":"user","content":"Explain how SSE works in 300 words."}]}' | tail -n 4

The last four lines should show "usage":{"prompt_tokens":...} and data: [DONE]. No usage means you estimate locally (roughly 1.5 tokens per CJK character, 1.3 per English word). No [DONE] and a closed socket means truncation — long jobs need chunking or non-streaming calls.

Reasoning models add a third trap: the thinking text arrives under reasoning_content, reasoning, or thinking depending on the implementation. Read all three keys rather than guessing one.

3. Multimodal: remote URLs not fetched, detail ignored#

image_url accepts two shapes: a public URL, or data:image/png;base64,.... The official API downloads URLs for you. Many compat layers don't — no egress, or simply unimplemented. You either get a 400, or the URL is passed through as text and the model starts guessing from the filename. The output reads fluently and is entirely fabricated, which is worse than an error.

detail: "high" is the second trap. The official API tiles high-resolution images; a layer that ignores the parameter downsamples to a fixed size, and small text — invoice numbers, stack traces, table cells — turns to mush.

You don't need code to check this. Take one screenshot with small text, ask "read the number on the third line", and run it twice with detail set to high and low. Identical answers mean the parameter is ignored; both wrong means resolution is already insufficient.

The engineering answer is to remove the uncertainty: inline base64, long edge capped at 1568px, detail stated explicitly. Then you don't depend on the gateway fetching URLs or honouring flags. File inputs (PDF, audio) diverge even more; the capability table in the multimodal docs is worth reading before you commit.

4. Structured output: json_schema downgraded#

response_format: {"type": "json_schema", ...} is a hard constraint. {"type": "json_object"} merely asks for JSON. A layer that only implements the latter gives you three failure modes: missing keys, extra keys, and the classic — a helpful ```json fence around the whole payload.

The problem is that failure is silent. json.loads may succeed while the payload is {"result": {...}} instead of your top-level fields, and the KeyError surfaces three call frames away.

The fix is boring: never trust json_schema to have taken effect. Validate with pydantic immediately, put the name of the missing field into a single retry prompt, and stop at one retry. That alone removes a large share of production 500s. Error-code and billing logic is in billing and error codes.

5. Prompt caching: the money you can't see#

Applications with long system prompts — support bots, code review, doc QA — are extremely sensitive to caching. Officially, cache reads bill at 0.1× input. A compat layer may not implement cache_control at all, so every request pays full price. No warning; you just see input costs at 3–5× the estimate.

There's exactly one way to know: look for the cache fields in the usage block.

u = resp["usage"]
print(u.get("prompt_tokens_details", {}).get("cached_tokens"))              # OpenAI shape
print(u.get("cache_read_input_tokens"), u.get("cache_creation_input_tokens"))  # Anthropic shape

All None or zero means budget as if there is no cache. One more detail people miss: Anthropic enforces a minimum cacheable length (roughly 1024 tokens on Sonnet and Opus). A short system prompt yields cache_creation_input_tokens: 0 forever — not broken, just below the threshold.

One probe script for all five#

Merge the snippets above, run it before and after the migration, and diff the output:

import json, os, requests
B, K = os.environ.get("ZAIMOD_BASE", "https://api.zaimod.com/v1"), os.environ["ZAIMOD_API_KEY"]
H = {"Authorization": f"Bearer {K}"}
def post(body, stream=False):
    return requests.post(f"{B}/chat/completions", headers=H, json=body,
                         timeout=90, stream=stream)

ok = True
# 1) tool calls
r = post({"model": "gpt-4o-mini", "messages": [{"role": "user",
    "content": "Get both weather and time for Hangzhou. Call both tools."}],
    "tools": TOOLS, "temperature": 0}).json()
n = len(r["choices"][0]["message"].get("tool_calls") or [])
print("tool_calls", n, "PASS" if n == 2 else "DEGRADED"); ok &= n == 2

# 2) streaming usage
buf = [c for c in post({"model": "gpt-4o-mini", "stream": True,
    "stream_options": {"include_usage": True},
    "messages": [{"role": "user", "content": "Count to twenty."}]}, stream=True).iter_lines()]
has_usage = any(b'"usage"' in x and b"prompt_tokens" in x for x in buf if x)
print("stream usage", "PASS" if has_usage else "MISSING"); ok &= has_usage
print("ALL PASS" if ok else "degraded — apply the workarounds above")

Multimodal and structured output resist automation: their failures are "wrong answer", not "missing field", and a script can't judge that. Run those two by hand.

What a full self-check costs#

Five probes at 600 input + 120 output tokens each: 3,000 input, 600 output. At live list prices (USD per 1M tokens, default group):

In other words: a complete pre-migration check on the cheapest model costs less than a tenth of a cent, and it surfaces in ten minutes what otherwise takes three days of production debugging. What's actually expensive is discovering a week later that caching never worked. Live numbers are on the pricing page; this table is only for order-of-magnitude.

Conclusion#

The right order isn't "change base_url → ship → debug", it's "change base_url → run the probes → record degradations → apply workarounds". Almost every degradation has a cheap workaround: base64 instead of URLs, pydantic instead of strict, local estimation instead of usage, explicit chunking instead of long streams. The only genuinely unfixable gaps are vendor-private endpoints — thinking blocks on /v1/messages, previous_response_id on /v1/responses — and those deserve a separate decision.

When choosing a provider, look past price at whether it documents its capability boundaries. That's the same trait as the five signals a gateway is about to fold, just viewed from the other side.

New accounts come with trial credit, so the probe script runs immediately. Decide your primary model after you know which capabilities survive, and pay as you go — balances don't expire.

Further reading#

• OpenAI-compatible protocol: what the translation layer really does
• OpenAI endpoint field reference
• Five signals an API gateway is about to fold