How Small Decision Models Make AI Products More Reliable

Before Datadog lets an agent run a shell command to fix a production issue, a second model looks at the command and answers one narrow question: how risky is this? It doesn't write an explanation or propose a better command. It returns a score, and the remediation either proceeds or stops. Datadog told OpenAI that this check runs 40x faster at p95 than the frontier models it compared against, with comparable accuracy, which is what made it possible to put the check in the hot path, the live request path a user or agent waits on (OpenAI, 2026).

That check is the part of the system that decides whether anything the agent wrote gets to touch production. The generative model gets the demo. The checker is what gets you to production. OpenAI decision models, and the wider category of decision models they belong to, make that layer cheap and fast enough to run on every output instead of a sample.

One naming note before going further, because search results blur it. OpenAI's product is the Decisions API, which runs on GPT-6 Luna. Jev is a different product: a purpose-built decision model from TypeSafe AI that launched on September 15, 2026, roughly two weeks before OpenAI announced its own. "OpenAI Jev" isn't a thing. The category both belong to is what this article is about, and the interesting part isn't either API. It's where the check sits, who sets its thresholds, and what happens when it says no.

What a decision model is and how it differs from a generative LLM

A decision model takes context plus a question whose answers you define in advance, and it returns a typed answer with a probability attached. It does not generate prose. You ask "Does this reply answer the user's question?" or "Which queue does this ticket belong in?" and you get back a number or one of your options, ready for code to branch on.

OpenAI's version is the Decisions API, served at POST /v1/decisions with gpt-6-luna as the only supported model. OpenAI made it generally available on October 22, 2026, after a public beta (OpenAI, 2026). It supports three answer types:

• Predicate: the probability, from 0 to 1, that a condition is true. "Does this product photo show visible damage?"
• Choice: one of the values you supplied, plus a probability distribution and a confidence field. "Continue, retry, or ask for clarification?"
• Score: a rating against 2 to 10 ordered rubric levels, returned as the probability-weighted average of the level indices, plus confidence. "How severe is this issue?"

Inputs can be plain text, images, native JSON, or a conversation history that includes tool calls and tool outputs. A single request can carry up to 400 questions, double the beta limit (OpenAI, 2026).

Side by side, the difference from a normal LLM call is plain.

OpenAI draws the same boundary in its own docs. If you need to generate an object that follows your schema, or a written explanation, use Structured Outputs or function calling through the Responses API. Use Decisions when the application needs a bounded answer (OpenAI, 2026).

TypeSafe describes Jev as a "System One" model, after Kahneman's fast mode of thinking, and that framing is a fair summary of the whole category. These models trade the ability to write for speed, low cost, and answers that always come back in the shape you asked for.

Why decision models matter for AI product teams

Most production AI systems already make dozens of small decisions per request. Is this input in scope? Did the retrieval step find the right passage? Is the draft safe to show? Teams usually answer those questions in one of two bad ways: they skip them, or they make another full generative call to answer each one. Decision models give you a third option.

Latency drops enough to change where checks can live. OpenAI says the Decisions API returns answers up to 10x faster than GPT-6 Luna through the Responses API (OpenAI, 2026). Customer reports in the same announcement are sharper. Sierra says it picks the next action in live customer conversations at under 280 ms p95, about 3x faster than its previous approach. Box says it got roughly 5x faster route selection on 30 certificate-of-insurance cases with the same accuracy as the Responses API.

Validation stops being the expensive part of your eval budget. The Decisions API costs $0.10 per million uncached input tokens and $0.01 per million cached input tokens, with no charge for output tokens (OpenAI, 2026). When your quality checks cost a fraction of the generation they're checking, you can afford to run them on every output rather than a 5% sample.

Answers are consistent in format. A choice question can only return one of your values. No regex to parse "I think this is probably acceptable." Your policy code stays simple, and your logs become queryable.

Validation scales with your product. Many questions per request, plus prompt caching for a shared prefix of at least 1,024 tokens, mean you can check one long document against twenty criteria without paying twenty times for the document (OpenAI, 2026).

None of this makes a decision model smarter than the model that generated the output. It makes the check cheap, and cheap checks are the only ones teams still run three months after launch. That matters for your operating cost too, because cost per accepted result is a better number to manage than total spend.

Practical use cases for AI output validation

The pattern underneath every use case is the same: one bounded question, a defined answer set, and an action for each branch. Here is how that plays out in the places we see it most often.

Checking an assistant's reply before the user sees it

A support assistant drafts a reply. Before it ships, a set of predicate questions run against the user's message and the draft. Does the reply answer what was asked? Does it promise a refund the policy doesn't allow? Does it include personal data that should not be there? Each returns a probability. Above your threshold, the reply goes out. Below it, the reply is regenerated or handed to a person. This is AI output validation without another expensive generative call per check.

Classifying and routing documents

Box uses the Decisions API with Box Extract to classify business documents and route each one to the right next step, sending lower-confidence cases to reviewers (OpenAI, 2026). A choice question with your departments or workflows as the options does the same job for intake forms, tickets, or contracts.

Verifying business rules against structured data

OpenAI's documentation includes a clean example: pass a support ticket as text and the order record as native JSON, then ask whether the ticket describes a duplicate charge that the order supports. That is a rule check that needs meaning (what the customer is claiming) and data (what the charges show) at the same time. A regex can't do it. A decision model can, in one call.

Moderation and policy checks

Predicate and score questions work well as a fast first pass on content: is this message off-topic, abusive, or outside the product's scope? Treat this as a layer, not a replacement. Hard policy that must hold every time belongs in deterministic code, and a decision model's job is the judgment calls around it.

Routing agent workflows

OpenAI's own example of a choice question is an agent deciding whether to continue, retry, or ask for clarification. The same shape handles tool-call approval: is this action within policy? Datadog's risk scoring of agent-generated shell commands is the production version of this, and it is why guardrails no longer have to be the slow part of an agent. If your agents are already scaling faster than their guardrails , this is where the gap closes.

Quality gates on generated artifacts

When a model writes requirements, specs, code, or test cases, a decision model can check each artifact against predefined criteria before a person spends time on it. That is the use case closest to our own work, and it is where evals come in.

Decision models inside your AI evals

An eval is a set of questions you ask about your system's outputs, repeated often enough that you notice when the answers change. Decision models fit evals naturally because most eval criteria are already bounded questions in disguise.

Scoring generated responses. A score question with defined levels (for example, "unusable," "needs edits," "acceptable," "ready to ship") gives you a rubric grade per output. Write each level's criteria precisely, because the model can only be as consistent as your rubric.

Detecting incorrect outputs. Break "is this correct?" into predicates you can answer one at a time. Does the summary contradict the source? Does the code call a function that isn't in the provided API surface? Does the test case have an expected result?

Verifying requirement compliance. With up to 400 questions in one request, you can ask one question per acceptance criterion against a single generated artifact and get a pass map back, not a vague overall grade.

Monitoring quality continuously. Run the same cheap checks on a sample of production traffic every day. Watch the distribution of probabilities, not just the pass rate. A slide in the average, say from 0.92 to 0.81, tells you something changed long before the pass rate drops. That kind of drift is exactly why a feature can pass every eval and still fail in production .

The probabilities are where most teams get careless. A 0.9 that turns out right only 70% of the time is worse than no number at all, because people trust it. So the evaluation method matters more than the model choice. Here is the method we use:

1. Build a labeled set from your own cases. Hexploits recommends two to five hundred real cases with known answers before choosing a vendor (Hexploits, 2026). Pull them from production logs, not from imagination.
2. Measure accuracy and calibration. Accuracy tells you how often the answer is right. Calibration tells you whether a 0.9 probability is right about 90% of the time. Hexploits reports that the first independent decision-model benchmark found every model overconfident on knowledge and reasoning tasks (Hexploits, 2026), so treat raw probabilities as a starting point.
3. Set thresholds by the cost of each mistake. OpenAI's guidance is to use labeled examples from your application to set thresholds for routing, filtering, or review, based on the cost of false positives and false negatives (OpenAI, 2026). A false pass on a refund promise costs more than a false flag on a typo.
4. Test order sensitivity on choice questions. Developers have reported that Jev's choice probabilities shift when the answer list is reordered (Firecrawl, 2026), and the same risk is worth checking on any decision model. Shuffle your options in testing and see whether the answer holds.
5. Re-run the set whenever anything changes. New question wording, new rubric levels, new model version. The labeled set is the asset. The thresholds are derived from it.

Testing non-deterministic systems is its own discipline, and testing AI features when outputs are non-deterministic covers the broader strategy. Decision models are one instrument inside that strategy, not the whole of it.

Our experience: AI output quick validation

At Advantage Works, the place this pattern earned its keep first was QA. We rebuilt test-case authoring around AI generation and brought one feature's test cases from about four hours of a QA engineer's time down to about forty-five minutes, measured on our own work. We wrote up the full test-case workflow separately. The speed came from generation. What made the speed safe was the validation around it.

The pipeline has a step we call quick validation. The model drafts candidate cases, each tagged to the acceptance criterion it exercises. Then an automated pass runs before any human looks at the output. It flags any criterion with zero cases, any duplicate or near-duplicate, and any case without an expected result. Human review comes after that, and it is aimed at the edge, negative, and boundary cases where a plausible-sounding step is most likely to be wrong.

That automated pass was deterministic: a coverage map and a dedupe check. It caught the mechanical failures cheaply, and it couldn't catch meaning. It could tell us a case existed for criterion 7. It couldn't tell us whether the case tested criterion 7. That question is exactly the bounded semantic judgment a decision model answers, and it is the layer we now design in between the deterministic checks and the human reviewer.

A few principles from that work now shape how we design validation for clients:

• Write each criterion as a question a reviewer could answer yes or no. If a senior engineer can't answer it consistently, a model won't either.
• Keep one criterion per question. Compound questions produce muddy probabilities and make failures hard to diagnose.
• Spend expensive judgment only on uncertain cases. Deterministic checks first, the cheap semantic check next, and the human or the stronger model only for what is left.
• Log every decision with its probability. The log becomes tomorrow's labeled set, and it's the audit trail you hand legal or a reviewer when someone asks why an output shipped.

The principle behind all of it is the same one that runs through shift-left testing for AI-generated code : when generation gets cheap, verification becomes the constraint, so verification has to get cheap too.

Architecture: generator, decision model, and policy

The architecture that works is three layers, and the separation between them is the point.

1. The generator (a general LLM) produces the output: a reply, a spec, a block of code, a set of test cases.
2. The decision model runs a set of bounded checks against predefined quality criteria and returns typed answers with probabilities.
3. The policy layer is your own code. It maps those answers and your thresholds to an action.

This is where a decision layer starts making workflow decisions, not just quality calls. For each output, the policy layer picks one of four paths:

• Accept when every check clears its threshold.
• Regenerate when a specific criterion fails, feeding the failed criterion back into the next generation so the retry is targeted.
• Escalate to a stronger model when the decision model is uncertain, so the expensive model only sees the hard cases.
• Route to human review when stakes are high or the output fails repeatedly.

Three design rules keep this from turning into a mess. First, decision models return answers and your application runs the actions, which OpenAI states plainly in its documentation (OpenAI, 2026). Keep policy in code you can test and version. Second, set a latency budget for any check in the request path, and keep slower checks asynchronous. Third, cap retries, because an endless regenerate loop is a quiet way to burn budget.

This is also how you get control over AI operating cost. Most of the spend in a naive system goes to sending every case to the most capable model. With a decision layer, the strong model sees only the uncertain band. Speak reported that the Decisions API integration in GPT-Live cut its time to next action from 1.4 seconds to 0.13 seconds (OpenAI, 2026). At that speed a check stops feeling like a cost inside the product. It also forces the question in whether you can prove your agent did the task correctly , because every decision now leaves a trace.

When to use a decision model vs LLM-as-a-judge, deterministic checks, or human review

Decision models don't replace the other validation methods. They fill the gap between them. We compare the four on six criteria: cost per check, latency, output type, ability to handle nuance, consistency, and whether you get an explanation.

LLM-as-a-judge is still the right tool when you need to know why an output failed, or when the criteria are open-ended. It is also how many teams first discover what their decision-model questions should be. Run the judge on a sample, read its reasoning, and turn the recurring failure reasons into bounded questions.

Deterministic validation should always run first. It's free and exact. If a JSON field is missing, you don't need a model to tell you.

Human review is the ground truth. Your labeled set comes from people, and the uncertain band in your policy layer should end with people when the stakes justify it.

The working order is simple: deterministic checks, then the decision model, then an LLM judge or a human for whatever remains uncertain. Each layer handles what it is good at and passes on less to the next, so the most expensive check runs on the fewest outputs.

Decision models are a poor fit in three situations. If the decision needs a written explanation for a user or an auditor, use a generative model. If the reasoning is open-ended and multi-step, a bounded question will flatten it. If a rule must hold every time without exception, encode it in code, not a probability.

Where to start

Production AI has moved past the question of whether a model can produce the output. The harder question is whether you can tell, cheaply and quickly, which outputs to trust. The generator writes the answer. The decision layer decides whether it counts. Decision models make that check affordable enough to run on everything, and the architecture around them (thresholds, routing, logging) turns AI quality from a hope into a number you can track.

The first step is small. Pick one recurring decision your product makes, write the question and its allowed answers, collect a few hundred labeled cases from real traffic, and set thresholds by what each mistake costs you. That gives you one measured validation layer, which is more than most AI products ship with.

If you want help deciding where that layer belongs in your product and what it should cost to run, that's what our Discovery Sprint is for. In one week, we map your AI workflows, identify the validation points that matter, and give you a concrete plan for making your outputs reliable and measurable.

Book a Discovery Sprint