Guardrails and Structured Output

LLM structured output guardrails are the contract between a stochastic model and a deterministic downstream system: the model must return data that fits a schema, and the runtime must enforce it, or the next stage breaks. Free-form text is for humans; production agents need parseable, typed output.

Three layers stack together:

1. Decoding-level constraints force the model to emit only tokens that satisfy a grammar or JSON schema. Examples: OpenAI JSON mode, Anthropic tool use, Outlines, Guidance, llama.cpp grammar.
2. Schema validation with Pydantic or zod runs after generation: types, ranges, enums, refusal flags.
3. Guardrail policies check content, not shape: PII, toxicity, jailbreak attempts, refusal handling.

flowchart LR
  P([Prompt + schema]) --> LLM
  LLM -->|JSON / tool call| Dec[Decoder constraint]
  Dec --> Val[Pydantic validate]
  Val -->|ok| Pol[Policy guardrails]
  Val -->|fail| Retry[Retry loop]
  Retry --> LLM
  Pol -->|ok| Out([Typed action])
  Pol -->|refuse| Block([Blocked response])

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class P client;
  class LLM compute;
  class Dec,Val,Pol service;
  class Retry queue;
  class Out client;
  class Block datastore;

Why guardrails matter: without them, a single malformed token from the model crashes the parser, and a single hallucinated field corrupts the database. With them, the model becomes a typed function in a regular program. The cost is small (10-30ms validation, occasional retry) and the reliability gain is the difference between a demo and a production system.

Problem statement

A support agent calls an LLM to extract a refund request from a customer email and write it to a payments API. The LLM sometimes returns extra prose around the JSON, sometimes invents fields, occasionally outputs a refusal. The downstream API has zero tolerance for bad input. Design the pipeline that turns an unreliable model into a typed function.

LLM structured output guardrails solve this by combining decoding-side constraints, schema validation, and policy checks into one pipeline.

Three layers of control

flowchart TB
  subgraph L1[Layer 1: decoding constraints]
    JSON[JSON mode]
    Tool[Tool / function call]
    Gram[Grammar / regex<br/>Outlines, Guidance]
  end
  subgraph L2[Layer 2: schema validation]
    Py[Pydantic / zod]
    Ref[Refusal detector]
  end
  subgraph L3[Layer 3: policy guardrails]
    Pii[PII redaction]
    Tox[Toxicity / safety]
    Inj[Injection classifier]
  end
  L1 --> L2 --> L3

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
  class JSON,Tool,Gram service;
  class Py,Ref compute;
  class Pii,Tox,Inj external;

Layer 1: decoding-level constraints

JSON mode

OpenAI, Anthropic, Gemini, and most local engines support a flag that biases the sampler to emit valid JSON. Useful as a baseline, but does not enforce a specific schema, only valid syntax.

Tool / function calling

The strongest provider-level guarantee. You pass a JSON schema; the model is biased to emit a tool call that matches it. With OpenAI's tool_choice: required and Anthropic's tool_choice: any, you get schema-valid output most of the time, plus a clear "I cannot answer" path via a refusal tool.

Grammar-constrained decoding

Libraries like Outlines, Guidance, lm-format-enforcer, and llama.cpp GBNF grammar enforce a context-free grammar at every token: any token that would violate the grammar gets its logit set to negative infinity. Result: 100% syntactically valid output, even on small local models.

# Illustrative Outlines example
from outlines import models, generate
from pydantic import BaseModel

class Refund(BaseModel):
    order_id: str
    amount_cents: int
    reason: str

model = models.transformers("meta-llama/Llama-3.1-8B-Instruct")
generator = generate.json(model, Refund)
refund = generator("Extract refund: 'Order 1234, $19.50, item broken'")
# refund is a typed Refund instance

Cost: decoding gets slightly slower (token-level mask computation). Benefit: parser never fails.

Layer 2: schema validation

Even with JSON mode, validate after generation. Pydantic gives types, ranges, enums, and custom validators in one place.

from pydantic import BaseModel, Field, field_validator
from typing import Literal

class Refund(BaseModel):
    order_id: str = Field(pattern=r"^ORD-\d{6}$")
    amount_cents: int = Field(ge=1, le=1_000_00)
    reason: Literal["damaged", "wrong_item", "late", "other"]
    customer_note: str = Field(max_length=500)

    @field_validator("reason")
    @classmethod
    def reason_lower(cls, v: str) -> str:
        return v.lower()

Failures should be structured, not exceptions thrown into the wild: log the raw model output, the violation, and the retry attempt count.

Refusal detection

Models sometimes refuse: "I cannot help with that." If your downstream system parses any JSON-looking object, a refusal that happens to include braces will silently break things. Make refusal a first-class branch:

class AgentResponse(BaseModel):
    refusal: str | None = None
    action: Refund | None = None

    @field_validator("action")
    @classmethod
    def one_of(cls, v, info):
        if (v is None) == (info.data.get("refusal") is None):
            raise ValueError("Set exactly one of action or refusal")
        return v

OpenAI's structured outputs already supports a top-level refusal field in the response object; treat it as a tri-state (ok, refused, error).

Retry-on-validation-failure

def call_with_retries(prompt: str, max_tries: int = 3) -> AgentResponse:
    last_error = None
    for attempt in range(max_tries):
        raw = llm_call(prompt, response_format=AgentResponse)
        try:
            return AgentResponse.model_validate_json(raw)
        except ValidationError as e:
            last_error = e
            prompt = (
                prompt
                + f"\n\nPrevious attempt failed validation: {e}\n"
                + "Return ONLY valid JSON matching the schema."
            )
    raise RuntimeError(f"Failed after {max_tries} retries: {last_error}")

Cap retries (2-3 is enough), back off if you are paying per call, and log every failure so you can spot model-drift regressions.

Layer 3: policy guardrails

Shape is necessary, not sufficient. A schema-valid response can still leak PII or contain a jailbreak echo. Run a second pass:

• PII redactor (Presidio, AWS Comprehend, local regex packs) scrubs SSN, credit cards, addresses.
• Toxicity / safety classifier (OpenAI moderation, Llama Guard, Perspective API) flags hate, self-harm, sexual content.
• Injection echo detector flags when the output contains hallmarks of a successful injection (e.g. the model started by repeating "Ignore previous instructions").
• Domain policies (e.g. financial disclaimers, medical refusal patterns) live as plain regex or LLM-as-judge checks.

Putting it together

sequenceDiagram
  participant App
  participant LLM
  participant Val as Validator
  participant Pol as Policy
  participant Down as Downstream API

  App->>LLM: prompt + schema (tool call)
  LLM-->>App: structured response
  App->>Val: validate(response, AgentResponse)
  alt invalid
    Val-->>App: ValidationError
    App->>LLM: retry with error feedback
    LLM-->>App: corrected response
    App->>Val: validate again
  end
  Val-->>App: typed object
  App->>Pol: scan(text)
  Pol-->>App: clean / flagged
  App->>Down: call with validated payload

Production failure modes

• Schema drift. Schema updated, prompt not. Always version the prompt with the schema, regenerate examples in the prompt from the schema.
• Silent truncation. Long outputs hit max_tokens mid-JSON; the parser fails. Set max_tokens based on schema size estimates, or use streaming with structured-output libraries that detect truncation.
• Provider differences. OpenAI tool calls, Anthropic tool use, Gemini function calls, and local grammar all diverge in edge cases (e.g. nested optional fields, enum casing). Abstract behind one adapter, eval all providers.
• Cost of retries. A 10% retry rate doubles average cost on a cached-prompt workload. Track retry rate as an SLO.
• Over-constrained grammar. Too-strict grammar makes the model unable to express "I cannot answer". Always include a refusal branch in the schema.

Quick reference

TL;DR

Constrain decoding, validate the schema, scan the content. Treat the LLM as a typed function.

Three layers

Layer	Tool	Catches
Decoding	JSON mode, tool calls, Outlines, Guidance	Syntax errors
Schema	Pydantic, zod	Wrong types, ranges, missing fields
Policy	Presidio, Llama Guard, moderation API	PII, toxicity, injection echoes

Decoding options

• JSON mode: valid JSON, not your schema.
• Tool / function call: provider-enforced schema (OpenAI, Anthropic).
• Outlines / Guidance / GBNF: token-level grammar mask, 100% syntactic compliance.

Pydantic essentials

• Use Literal[...] for enums.
• Field(ge=, le=, pattern=) for ranges and regex.
• @field_validator for custom rules.
• model_validate_json(raw) for parsing.

Refusal pattern

• Top-level refusal: str | None.
• Tri-state: ok | refused | error.
• Never collapse refusal into empty payload.

Retry loop

• Max 2-3 attempts.
• Feed validation error back into prompt.
• Log raw output + violation + attempt count.
• Cap retry rate as an SLO.

Policy guardrails

• PII redaction (Presidio).
• Safety classifier (Llama Guard, OpenAI moderation).
• Injection echo detector.
• Domain regex (medical, financial).

Watch-outs

• Schema drift, version prompt with schema.
• Truncation mid-JSON, set max_tokens from schema size.
• Provider quirks on nested / optional fields.
• Cost: retry rate ~10% doubles spend.
• Over-strict grammar blocks "I cannot answer".

Refs

• Outlines (dottxt-ai)
• Guidance (Microsoft)
• OpenAI Structured Outputs guide
• Anthropic Tool Use docs
• Pydantic docs
• Llama Guard (Meta)

FAQ

How do you force an LLM to return valid JSON?

Use the provider JSON mode or tool use API, validate the output with Pydantic or zod, and retry once with the validation error appended to the prompt. Grammar-constrained decoders guarantee shape at the token level.

What are LLM guardrails?

Guardrails are policies that check LLM input and output beyond shape: PII detection, toxicity filters, jailbreak detection, refusal handling, and topic restrictions. They run before and after the model call.

Should I use JSON mode or function calling?

Function calling, also called tool use, is preferred when the LLM should pick from a set of typed actions. Plain JSON mode is fine when you just need a single structured response without action routing.

What is grammar-constrained decoding?

It restricts the next-token sampler to tokens that satisfy a context-free grammar or JSON schema. Tools like Outlines, Guidance, and llama.cpp grammars guarantee parseable output without retries.

How do you handle LLM output validation failures?

Run a bounded retry loop with the Pydantic error and the offending output fed back to the model. Cap at two retries, log the failure, and surface a typed error if the schema cannot be satisfied.

Related Topics

• Prompt Injection Defense: policy guardrails catch echoes of successful injections
• LLM Evals: regression-test schema conformance and refusal handling
• Tool Use and Function Calling: provider-level structured output mechanism