Receiving AI's answer as a structure
I understand the approach of receiving AI's answers in a predetermined structure rather than free-form sentences, and the perspective of designing even for failure.
Why JSON Output Breaks Even When You Ask for It
Sometimes you ask an AI for JSON, but the result doesn't parse immediately.
There's a "Of course!" at the beginning, the JSON is wrapped in a markdown code block, and a sentence like "Let me know if you need anything else" comes at the end. It's a friendly answer to humans, but a failed output to JSON.parse().
Why JSON Breaks Even When You Ask for It
The problem isn't that the AI didn't understand your instruction.
The AI understood "Give me JSON." So it provided JSON. It just added natural sentences before and after it. From the AI's perspective, that felt like a more complete answer.
The failure patterns typically come in three varieties: preamble before the JSON, explanatory text after the JSON, or the JSON itself being grammatically malformed — missing quotes around keys, trailing commas after the last item, or single quotes mixed in. In any case, the parser throws an error.
It's easy to think "better instructions will solve this," but sometimes that alone isn't enough.
AI is Trained to Speak in Natural Language
LLMs are fundamentally trained to produce answers that humans can read. When given a question, rather than simply returning a value, they're likely to explain the context, add friendly sentences, and continue answering in a way that looks polished.
This is because most of the training data was natural language text. Phrases like "Of course!", "Below is", "I hope this analysis was helpful" — these expressions appeared repeatedly across countless question-answer pairs. JSON occupies a much smaller share in that data.
Structured output is about narrowing that natural way of speaking. Sometimes a single instruction like "Give me JSON" doesn't narrow that direction enough.
How to Narrow Possibilities with Prompts
It's often more effective to specify what not to do than to instruct what to do.
Output Rules:
- Output only pure JSON that starts with { and ends with }
- Do not write preambles or closing remarks like "Of course!", "Below is", "Additionally"
- Do not use markdown code blocks (```)
- Do not include comments
You can also separate output areas. If the model wants to write explanations, divide it into an explanation area and a final output area, keeping the value that the system actually reads only in the last output block.
The response must follow only the format below.
[OUTPUT]
{
"title": "",
"summary": "",
"tags": []
}
Do not write any sentences outside [OUTPUT].
One correct example is stronger than describing format in words. Show two or three completed input-output examples like few-shot, and the model naturally follows the pattern of starting directly with JSON without preambles.
These methods lower the failure rate but don't reduce it to zero.
When APIs Guarantee Format
There are limits to matching format with prompts alone. That's why real services sometimes enforce structure at the API level.
OpenAI's Structured Outputs specifies the response shape through JSON Schema. When you use strict: true, you can force the model to generate output that matches the provided schema only.
Anthropic's Tool Use works slightly differently. When developers define a tool's input schema and use tool_choice to force the use of a specific tool, the model returns a structured response matching that schema instead of natural language. It's closer to a structure where the developer makes the model always answer in that format, rather than the model deciding to call a tool.
What matters isn't the specific API name. It's that you may need mechanisms outside the prompt to guarantee format.
Designing for Failure
Even with guaranteed format, content can still fail.
Fields might be missing, types might differ from expectations, or there might be insufficient input information for the model to fill values. When the AI fills in plausible values in such situations, parsing succeeds but the data becomes semantically wrong.
It's different when you build failure into the structure.
{
"answer": null,
"error": "missing_required_information",
"missing_fields": ["target_audience", "source_text"],
"confidence": "low"
}
Instead of having AI cover up what it doesn't know with plausible sentences, make it return failure as a value. You can re-request when parsing fails, or handle differently depending on failure type. Error becomes part of the design, not an exception.
What It Means to Trust Output
Structured output isn't decoration to prettily organize the AI's answer.
If the AI's answer needs to pass to the next piece of code, that answer must be a value, not a sentence. Values have position, type, and defined form when they fail. From that point on, the AI becomes part of the system, not just a conversation partner.
Trusting output doesn't mean believing the AI unconditionally. It means making it answer in a form you can trust.