Yes/no decisions for agents

What the number actually is, and what it is not

Under the three types is one operation: natural-language inference. The model is given your state as a premise and your question as a hypothesis, and it returns how strongly the premise entails the hypothesis. A noul is that score directly.

For a choice, every option is scored the same way and the scores are then divided by their total so they sum to one. That last step is the part worth understanding, because it throws information away. Three options that each score 0.1 — the model finding none of them supported — come back as an even 0.33 / 0.33 / 0.33. Two options scoring 0.9 and 0.85 — both strongly supported — come back as 0.51 / 0.49. Those are opposite situations and they look identical.

So the number beside a choice is a share of the agreement the model found, not the probability that the answer is right. It ranks the options against each other honestly and says nothing about whether any of them fit. Treat it as an ordering with a margin: a wide gap means the model preferred one option clearly, a narrow one means it did not, and neither tells you the answer is correct.

This is where a model trained for calibrated decisions differs, and it is the concrete version of the warning above. A calibrated model's probabilities are fitted against real outcomes, so across many predictions the ones it calls 0.7 come true about seventy percent of the time. Nothing here has been fitted that way. Pick your threshold by running your own labelled examples through it and looking at where the errors land — not by reading the number as a percentage.

Where a judgment sits in a real workflow

It is rarely the decision on its own. The shape that works is judgment for the part a program cannot read, and ordinary code for everything else:

1. Build the state: the customer's message, the transactions it refers to, the policy that governs it.
2. Ask the independent questions together in one call — was a refund requested, does the evidence read like a duplicate charge, does the tone suggest escalation.
3. Combine the answers with deterministic checks your code already has: the amount, the account age, whether a refund was issued before.
4. Route. Act automatically where the margin is wide and the checks agree, send the rest to a person or to a reasoning model.

Step four is the reason the margin is in the response at all. A judgment API that only returned labels would force you to act on every one of them; the margin is what lets you hand the narrow cases to something slower and better, which on this platform is a reasoning model one endpoint over.

The default backend is called OpenJev. It is an open-source NLI cross-encoder, MIT licensed, that we host ourselves. It is not Jev, it is not made by the people who make Jev, and it is not a smaller version of it. The name is the open model's own.

We expect it to be materially weaker than Jev, and we have not benchmarked the two against each other — so we are not going to put a number on the gap. What we can say is what the open model reports on standard natural-language-inference sets, which is the task underneath all three question types above, and nothing about how it does on yours. Try it on your own data before you depend on it.

What OpenJev actually is — the architecture, the licence and its authors' own benchmark figures — and how it compares to Jev are each their own page. Jev itself, from TypeSafe, is a different product that we do not resell and do not serve. There is one backend here and the response header names it, so you are never quietly handed something other than what you asked for. If Jev is what you want, go to TypeSafe for it.

One call: the state, the questions, the answers

curl -X POST https://api.blockrun.ai/v1/decide \
  -H 'authorization: Bearer $BLOCKRUN_API_KEY' \
  -H 'content-type: application/json' \
  -d '{
    "state": "Help! My payouts have been failing for 3 days.",
    "questions": {
      "is_urgent":   { "type": "noul",   "instructions": "Does this convey urgency?" },
      "department":  { "type": "choice", "instructions": "Which team should handle this?",
                       "criteria": { "billing": "Payments, refunds",
                                     "technical": "Bugs, outages" } },
      "frustration": { "type": "score",  "instructions": "How frustrated is the customer?",
                       "criteria": ["Calm", "Frustrated", "Very angry"] }
    }
  }'

# Free with any registered key. No payment header, no wallet.

state takes a string, an object or an array — a whole ticket, a diff, a tool result. The response carries the answers, the probabilities, and a header naming which backend served it. The key comes from user.blockrun.ai and costs nothing.

What this is not

It does not write, summarise, or explain. There is no free-text field in the response and there is not going to be: the whole point is an answer with a type and a number, which is the part a program can act on without a parser. For prose, the chat models are next door and speak the OpenAI protocol.

It also has no memory between calls, and it judges what you send rather than going to look anything up. If the judgment needs a fact from the web, fetch it with search and pass it in as part of the state.