Your model returns invalid JSON because response_format is not one feature
You set {"type":"json_object"}, it worked against GPT, you swapped the model string, and now you are stripping backticks in production. The parameter did not stop working. It never meant the same thing on the model you moved to.
The same parameter, three behaviours
| Backend | What the parameter is | Can the reply still be invalid? |
|---|---|---|
| OpenAI, DeepSeek, Z.AI | Forwarded to the provider. The shape is enforced while the tokens are chosen. | No, for the shape it enforces |
| Anthropic, Bedrock, Google | These APIs have no equivalent. A gateway (ours, or whatever you are using) appends a sentence to the system prompt and cleans up the reply. | Yes. It is a request, not a rule |
Nothing in the response tells you which one you got. Same field, same status code, same shape — and on one of those rows a retry loop and a try/catch are load-bearing.
What that looks like, measured
Same prompt — "Give me an object with keys city and population for Tokyo" — through one gateway on 2026-09-20, changing only the model string.
```javascript
{ city: "Tokyo", population: 37400068 }
```Fenced, and the keys are unquoted. That is a JavaScript object literal, not JSON — JSON.parse rejects it twice over.
```json
{ "city": "Tokyo", "population": 37400068 }
```Keys quoted now, and the input grew from 23 to 47 tokens — the instruction being appended, visible in the bill. The content became valid JSON. The fence did not go away on its own; something downstream has to remove it.
Here is the object in JSON format:
```json
{ "city": "Tokyo", "population": 14180000 }
```
*(Note: The population of ~14.18 million reflects the Tokyo
Metropolis as of recent official estimates…)*Prose in front, a parenthetical behind. The parameter was being dropped outright on that path — not forwarded, not emulated. This is what "supported" looks like when nobody checked.
Two of those were our bugs
The Gemini reply above is not a story about Google. It is our routing code: the Gemini path had no handling for response_format at all, while our own documentation claimed it was emulated there. Fixed the day this page was written.
The second one was worse because it was quieter. json_schema — the stricter request, the one you reach for when the shape actually matters — was silently dropped on Anthropic and Bedrock. Four separate places in the code tested for json_object and nothing else, and the parameter never reached the provider, so there was nothing upstream to reject it either. You asked for a guaranteed shape, got prose, and no error was ever raised.
Both are fixed and both are written into the API reference with the date. They are here because a page comparing how backends behave, written by the people who run the router, is worth nothing unless it can be caught naming its own defects.
What to actually do
- Find out which row you are on before you trust anything. Send your real payload to the model you actually ship, twenty times, and count the parses that fail. That number is the only one that matters, and it is per model, not per vendor.
- Parse defensively on the emulated row. Strip a leading and trailing fence, then try/catch, then one retry. Cheap, and it costs nothing on the native row where it never fires.
- Keep the schema small. On the emulated row the schema is a paragraph in the prompt: every field is tokens you pay for and attention the model spends on bookkeeping rather than the answer.
- If the shape is the whole point, pick for it. A model with native structured output on a boring task beats a smarter model you have to nurse. And when the output is one label, a number, or a choice from a list, a judgment model returns it typed with no parsing at all — see Decide, which is free, or the text classification API if the output is one of a set of labels.
Why one endpoint makes this checkable
Every measurement above is the same request with one string changed. That is the only reason they are comparable — three vendor accounts, three SDKs and three sets of defaults would have produced three anecdotes instead. Point your client at one base URL, change the model, and run your own version of this table against your own payload.
Structured output questions: JSON mode, schemas, which models enforce it
- Why does my LLM return invalid JSON even with JSON mode on?
- Because on several providers JSON mode is not a constraint. OpenAI-compatible backends enforce the shape while the tokens are chosen; Anthropic, Bedrock and Google have no equivalent parameter, so a gateway appends an instruction to the system prompt and tidies the reply. An instruction can be ignored. Nothing in the response tells you which of the two you got.
- What is the difference between json_object and json_schema?
- json_object asks for any valid JSON. json_schema asks for a specific shape and, on a backend that supports it natively, the decoder holds the model to it. On an emulated backend the schema becomes a paragraph in the prompt, which is a request rather than a rule.
- Which models actually enforce a schema?
- The ones whose own API takes the parameter — OpenAI and the OpenAI-compatible providers. Everywhere else it is emulated. The honest way to find out for the model you ship is to send your real payload twenty times and count the parses that fail; the answer is per model, not per vendor.
- JSON mode vs function calling — which should I use for structured output?
- Use whichever the model you picked actually enforces, and measure rather than assume. The deeper point is the same either way: a parameter being accepted is not the same as a shape being guaranteed, and only your own failure count tells them apart.
- Do I still need a try/catch around JSON parsing?
- On the emulated backends, yes, plus a fence strip and one retry. On the native ones it never fires, so it costs nothing to keep. Code that runs across several models needs it unconditionally.
- Does a bigger schema make structured output more reliable?
- On an emulated backend it does the opposite. The schema is text in the prompt: every field is tokens you pay for and attention spent on bookkeeping instead of the answer. Keep it to the fields you will actually read.
- Can I get structured output from Claude or Gemini at all?
- Yes, through emulation, and it works well in practice — the instruction lands and the reply is usually clean. What you do not get is a guarantee, so treat a parse failure as a normal outcome and handle it rather than as a bug to report.
- Is there a way to skip JSON parsing entirely?
- When the output is one label, one number, or one choice from a list, yes: a judgment endpoint returns the value typed with nothing to parse. That is a different tool from a chat model and it does not write prose, which is the point.
- How do I compare structured output across models fairly?
- Change one thing. Same payload, same request shape, one base URL, and only the model string differs — otherwise you are comparing three SDKs and three sets of defaults and calling it a model comparison.