How to Make Sure an LLM Returns JSON That Strictly Matches the Expected Format

01Core Conclusions

• JSON mode ≠ a schema guarantee. Across every vendor's semantics, response_format: json_object only guarantees that "the output is syntactically valid JSON" — it makes no guarantee whatsoever that the output conforms to any schema. OpenAI's own words: "Always use Structured Outputs instead of JSON mode whenever possible."
• The real mechanism-level guarantee comes from constrained decoding. A JSON Schema is compiled into a grammar, and during sampling each illegal token is masked out one by one — OpenAI Structured Outputs, Anthropic structured outputs, Gemini responseSchema, and Volcano Ark's (火山方舟) json_schema strict all fall into this category. Pure prompt steering scored only <40% compliance in OpenAI's internal evals, pure training improvements got to 93%, and only with constrained decoding does it reach 100% (structural validity only).
• Even the strongest strict mode still has three classes of residual failure: max_tokens truncation, safety refusals (the refusal text takes priority over the schema), and structurally valid but semantically wrong field values. The widely repeated claim that "once strict is on you don't need validation or retries" simply doesn't hold up when you check it line by line against four vendors' docs (§04 has the detailed accounting).
• Application-layer validation + retry + fallback is not community folklore — it's the vendors' own prescription. The official docs of Alibaba Cloud, Zhipu, and Ark all state explicitly: before you pass the output downstream you must do schema validation, and on failure you retry or switch models to repair it, and keep a simplified backup schema on hand.
• Domestic vendors are clearly diverging: Volcano Ark already supports json_schema + strict:true (beta, officially recommended but flagged as "use with caution in production"); the official APIs of DeepSeek, Tongyi Qianwen, and Zhipu GLM to this day offer only json_object, so the schema can only be written into the prompt and compliance rests on the model's goodwill.
• Conclusion: no single layer can be trusted on its own. Business-grade stability = request-layer constraint (turn on strict wherever you can) × response gate (truncation/empty-content detection + Zod validation) × recovery layer (repair → retry with error feedback → fallback) × observability layer (failure-rate metrics + alerting).

02Three Tiers of Guarantee Strength & How Constrained Decoding Works

Every approach on the market sorts into three tiers by "where the guarantee comes from," and the difference in strength is mechanistic, not a matter of degree.

How constrained decoding works

OpenAI's public description: it converts the submitted JSON Schema into a context-free grammar (CFG), and at each inference step uses the grammar to compute "the set of tokens legal at the current position," setting the sampling probability of illegal tokens to 0. The reason for choosing a CFG over a regex/FSM is that an FSM can't express recursive structures (matching brackets in deep nesting), whereas a CFG can support $ref: # recursive schemas. OpenAI officially credited open-source projects such as outlines, jsonformer, instructor, guidance, and lark as sources of inspiration.

The engineering cost is first-compile latency: the first request for a new schema requires preprocessing (OpenAI's 2024 figures: within 10 seconds for a typical schema, up to 1 minute for a complex one), after which cached reuse adds zero latency. Anthropic caches its compiled grammar artifact for 24 hours from last use, and the cache is invalidated as soon as the schema structure or the in-request tool set changes — meaning you should be restrained about schema versioning, because frequently changing the schema means continually paying the first-compile latency.

03Vendor Capability Comparison

All conclusions are based on the vendors' first-party docs, fetched live on 2026-07-07, with each item verified by 3-vote adversarial checking. Capabilities claimed by domestic vendors' third-party sites (SEO redirect pages / API resellers) were all re-verified against the official docs as the source of truth.

04Residual Failures That Even strict Mode Can't Escape

This is a conclusion consistent across all four of OpenAI / Anthropic / Google / Ark, and all four docs provide a corresponding detection method — meaning these branches must be written for every vendor.

05Self-Hosted Constrained Decoding

When you deploy an open-source model yourself, the schema-enforcement capability comes from the inference framework, not the model. This path can upgrade json_object-only open weights (Qwen, GLM, DeepSeek distillations) into grammar-level enforcement.

• Outlines (dottxt): the pioneer of the FSM/regex route and one of the inspirations OpenAI officially credited; a Python-ecosystem tool, and also the main opposing side in the "constrained decoding doesn't hurt quality" debate (see §10).
• XGrammar: a byte-level pushdown automaton; its core optimization is splitting the vocabulary into "context-free tokens (precomputable)" and "context-dependent tokens (checked at runtime)" — for Llama-3.1's 128k vocabulary paired with a JSON grammar, the latter number just 1,134 (<1%), so an adaptive mask cache can eliminate the vast majority of the overhead. It claims up to a 100× reduction in masking latency over prior approaches and up to 80× end-to-end speedup on an H100. Already integrated into the major open-source inference stacks.
• LLGuidance: no precompilation required, suited to dynamic/complex schemas (on complex schemas XGrammar times out while LLGuidance has zero timeouts).
• vLLM / SGLang: vLLM has removed the deprecated guided_json in v0.12.0, and the current parameter is structured_outputs; in the v0.10 era, enabling constrained decoding at batch ≥ 8 caused a significant performance drop, whereas SGLang overlaps CPU-side mask generation with GPU inference to nearly hide the overhead — how the framework integrates it is the decisive factor in self-hosting selection.
• llama.cpp GBNF: the grammar-constraint solution for local small models, with hand-written BNF grammars.

06The TypeScript Application-Layer Ecosystem

Four representative solutions, each embodying one line of thinking: SDK abstraction, retry loop, fault-tolerant parsing, and types-as-schema.

Vercel AI SDK — the SDK abstraction layer

• The developer provides a Zod / Valibot / JSON Schema, and the SDK uses it to both steer generation and validate afterward, returning a type-safe object; the current recommended API is the output property on generateText/streamText (Output.object/array/choice/json), abstracting away the differences between vendors' structured outputs (generateObject is no longer the docs' primary recommendation).
• On failure it throws AI_NoObjectGeneratedError, carrying the raw text, response metadata, token usage, and the underlying cause — a ready-made monitoring hook.
• Streaming pitfall: in streaming mode the partial object emitted mid-stream is not schema-validated; only the final complete object is guaranteed; the array elementStream is the exception — each element is validated individually once complete.

Instructor-JS — a retry loop with error feedback

• It attaches a Zod schema as response_model on chat.completions.create, going through function calling under the hood (TOOLS mode, the default recommendation) rather than a naked JSON prompt.
• The core is max_retries: on validation failure it automatically feeds the Zod validation error back into the next request for the model to fix, until it passes — including refine custom business rules. This is the benchmark implementation of the "validate-then-retry-with-error-feedback" pattern.
• The docs explicitly do not recommend the MD_JSON route (having the model emit JSON inside markdown), calling it unreliable and reserved only for vision models.

BAML (Schema-Aligned Parsing) — the fault-tolerant parsing route

• The idea is inverted: don't constrain the model, constrain the parser. SAP is a schema-aware fault-tolerant parser (edit-distance-style correction + schema-weighted cost) that directly fixes comments, unquoted keys, trailing commas, unescaped quotes, wrapping filler text, and extra fields, and can also complete partial objects to support streaming. Postel's law: "be liberal in what you accept."
• Berkeley Function Calling Leaderboard (n=1000 per model): SAP scores >90% on most models, generally beating native function calling — the most dramatic being gpt-4o-mini: FC 19.8% vs SAP 92.4% (claude-3-haiku 57.3→91.7, gpt-4o 87.4→93).
• BAML's field observation: on larger datasets, vendors' JSON mode often has a schema-level error rate above 10% — "structured outputs create a false sense of security."

TypeChat (Microsoft) — the pioneer of types-as-schema

It uses TypeScript type definitions as the schema, and on validation failure automatically repairs and retries carrying the type errors. Engineering-wise it has been surpassed by the ones above, but the paradigm of "the type system as the contract + a validate-and-repair loop" originates here.

07Parsing & Repair Strategies

Before re-calling the model (expensive, slow), deterministic repair (fast, free) should come first.

• Fence stripping: the model loves to wrap output in ```json ... ```, so strip it before parsing; a more robust approach is to take the slice from the first { to the last }.
• jsonrepair: deterministically fixes the most typical bad JSON from LLM output — truncation, markdown fences, missing quotes/commas/closing braces, single quotes, Python constants (None/True/False) — without another model call. The failure contract is clean: it either returns repaired JSON or throws a typed JSONRepairError, which you can wire straight into a fallback branch. There's a streaming variant, jsonrepairTransform (default 64KB buffer; a single overly long string triggers Index out of range, so bump the buffer for long text fields).
• partial-json parsing: in streaming-UI scenarios, do a best-effort parse of the unclosed JSON, used together with the "don't validate partial, only validate complete" discipline (see the Vercel AI SDK streaming pitfall).
• The cure for truncation is budget, not repair: max_tokens should leave headroom for the worst case of your output schema; repairing truncated JSON should be only the last resort — finish_reason detection + a reasonable budget is the real fix.

08Prompt-Layer Techniques

Even with strict on, don't delete this layer: it determines the quality of the "values," and it's your portability safeguard when migrating to json_object-only vendors.

• The "json" keyword is a hard constraint, not superstition: both DeepSeek and Alibaba Cloud Bailian require the string "json" to appear in the System/User message — Bailian returns an outright 400 if it's missing (messages must contain the word json in some form). Treat this as an API contract when writing prompt templates.
• schema + one concrete example: DeepSeek officially requires providing an example of the expected JSON; an example holds down format drift better than a schema description does, especially for weak models.
• Assistant prefill: prefill the assistant turn to start with {, forcing the model to continue from inside the JSON and naturally eliminating opening filler (a classic Anthropic technique; note that on Anthropic, prefill is incompatible with its structured output feature — pick one).
• Field order is chain-of-thought: generation is sequential — putting the reasoning field before the conclusion fields amounts to forcing "think first, answer later"; conversely, putting the answer first turns the reasoning field into an after-the-fact fabrication.
• Stop sequences: in non-JSON-mode scenarios, use a stop sequence to cut off the padding that follows the JSON.
• Don't use markdown-JSON: "please return JSON in a code block" is a pattern Instructor's docs explicitly do not recommend, and its parsing surface is the most fragile.

09Production Reliability Engineering

The recovery ladder (cheapest first)

1. Deterministic repair: fence stripping → jsonrepair → parse again. Zero cost, zero latency.
2. Retry as-is: the same request + exponential backoff with jitter (reference implementation: up to 3 times). Many failures are transient and clear up on the second try.
3. Corrective retry: feed the failed output and the precise parser/Zod error message back into the next request, restate the schema, and let the model fix its mistake (the Instructor pattern). A finite number of times, usually 1–2.
4. Fallback: switch to a simplified backup schema (Zhipu's official prescription) / switch to another model to repair the output (Alibaba Cloud's official prescription).
5. Business safety net: the last good result (cache), default content, or silently hiding the feature — never dump a 500 on the user.

Observability

• Record per endpoint × schema version: parse failure rate, truncation rate (share of finish_reason=length), empty-content rate, retry rate, final fallback rate, and alert on sudden shifts in the failure rate.
• On failure you must retain the raw output (otherwise you can't attribute whether it was model drift or a schema change).
• Evals have a hidden trap: when testing "accuracy on unstructured output," the result is highly sensitive to the parser — for the same batch of outputs, a hand-written lenient regex parser gives 61% vs an AI parser's 57%. Regression tests must fix the parser.

Parameters and versions

• Use low temperature for extraction/classification tasks; for generative tasks that need flair, don't sacrifice temperature for format — leave format to the constraint/validation layer.
• Put the schema version number into the cache key and monitoring dimension (once the schema changes, the vendor-side grammar-compile cache is invalidated and the first request incurs compile latency — warm it up once at release time).
• A model upgrade = the schema-compliance behavior may change, so run a fixed test suite for regression before switching models.

10Frontiers & Debates

• The "constraint tax" debate: arXiv 2408.02442 (Let Me Speak Freely?) claims format constraints hurt reasoning ability and circulated widely; dottxt's rebuttal, Say What You Mean, uses controlled experiments to show that when the prompt is written well, constrained decoding is no weaker than — and even outperforms — free generation. The current engineering consensus resolves it in two stages: reason freely first, then structure-extract, or use field order to pack the reasoning inside the schema (§08).
• Parser route vs constraint route: BAML's SAP demonstrates that "lenient parser + post-hoc validation" can match or even beat constrained decoding on accuracy, without depending on vendor capabilities and being model-agnostic — an important alternative for businesses running a mix of multiple vendors / domestic models.
• Dual-model / self-repair: "a generation model + a cheap small model to validate and repair" has graduated from folk trick to official vendor advice (Alibaba Cloud: "use another model to repair the output"). Semantic-level validation (LLM-as-judge spot-checking value correctness) is the next layer above schema validation, used to counter "structurally right but wrong in content."
• Trend read: constrained decoding is shifting from a differentiating capability to infrastructure (XGrammar entering mainstream inference stacks, vendors natively supporting it one after another); application-layer differentiation will move to value-level validation, evals, and observability.

11A Layered-Defense Reference Architecture

Each layer only catches the failure modes the previous layer clearly missed; remove any layer, and that layer's corresponding failures reach the user directly.

Minimal implementation skeleton (Bun / TypeScript)

async function callStructured<T>(schema: z.ZodType<T>, body: ChatBody): Promise<T> {
  for (let attempt = 0; attempt <= MAX_RETRY; attempt++) {
    const res  = await chat(body);                          // L1: json_schema strict (Ark beta, auto-falls back to json_object on failure)
    const choice = res.choices?.[0];
    if (choice?.finish_reason === "length") { metric("llm_truncated"); continue; }   // L2: truncation gate
    const raw = choice?.message?.content ?? "";
    if (!raw.trim()) { metric("llm_empty"); continue; }                              // L2: empty content (DeepSeek known issue)

    let parsed: unknown;
    try { parsed = JSON.parse(stripFences(raw)); }
    catch { try { parsed = JSON.parse(jsonrepair(raw)); }                            // L3: deterministic repair
            catch { metric("llm_unparseable", { raw }); continue; } }

    const v = schema.safeParse(parsed);                                              // L2: schema + refine business rules
    if (v.success) return v.data;
    body = withErrorFeedback(body, raw, v.error);                                    // L3: corrective retry (Instructor pattern)
    metric("llm_schema_fail", { issues: v.error.issues });
  }
  throw new StructuredOutputError();                                                // caller wires L3 safety net: cache last-good / degrade & hide
}

12Primary Sources

• OpenAI — Structured Outputs guide · developers.openai.com/api/docs/guides/structured-outputs; launch blog · openai.com/index/introducing-structured-outputs-in-the-api
• Anthropic — Structured Outputs · platform.claude.com/docs/en/build-with-claude/structured-outputs
• Google — Gemini Structured Output · ai.google.dev/gemini-api/docs/structured-output (updated 2026-07-06)
• Volcano Ark — Structured output (json_schema) · volcengine.com/docs/82379/1568221 (updated 2026-06-27), 82379/1958523
• DeepSeek — JSON Mode · api-docs.deepseek.com/guides/json_mode; Chat API reference
• Alibaba Cloud Bailian — Qwen structured output · help.aliyun.com/zh/model-studio/qwen-structured-output; DashScope API reference
• Zhipu — Structured output · docs.z.ai/guides/capabilities/struct-output, docs.bigmodel.cn/cn/guide/capabilities/struct-output
• XGrammar paper/blog, LLGuidance benchmarks, vLLM v0.12 release notes, SGLang docs
• BAML — Schema-Aligned Parsing and the BFCL benchmark; Instructor-JS docs; Vercel AI SDK docs; jsonrepair (josdejong)
• dottxt — "Say What You Mean" (a rebuttal to arXiv 2408.02442)

All vendor-capability conclusions were verified against official documentation via live fetch on 2026-07-07; third-party claims (such as SEO sites saying "DeepSeek/GLM already support json_schema") were verified to be false or non-official capabilities.