Make unsure models sure.

Feedback loops for a better classifier.

You can't improve a classifier without knowing where it goes wrong. Use our decide, feedback, and prototype APIs to build the best possible classifier.

1. 01

   Classify on Jev

   A customer has paid, but their account is still locked. Jev classifies the ticket as urgent and routes it to billing. Its confidence in that team is only 54%.

   Three fields round-trip in 111 ms median from a laptop. Unsure records every answer and its confidence under one decision_id, so this uncertain route can be reviewed.

   class Triage(BaseModel):
       urgent: bool
       team: Literal["billing", "support", "sales"]
       severity: Annotated[float, Score("Low", "Mid", "High")]
   
   decision = client.decide(
       prompt=(
           "Payment questions go to billing. "
           f"Ticket: {ticket.text}"
       ),
       schema=Triage,
   )
   ticket.decision_id = decision.id

   inbox · support@example.comticket #1042

   MR

   Maya Ruiz <maya@northwind.co>

   9:41 AM

   to support@example.com

   Paid, but still locked out

   Hi there,

   I paid the invoice, but my account is still locked. The receipt came through an hour ago and I have a demo at noon.

   — Maya

   client.decide · 111 ms

   decision3 fields · classified

   • urgenttrue0.92
   • teambilling0.54
     Review the team. Billing is the least certain answer, at 54% confidence.
   • severity1.7 / 20.85
2. 02

   Correct the route

   The review queue surfaces the least certain decisions. On this ticket, a human reviewer sees that payment succeeded, and restoring access belongs to support. They correct the team from billing to support.

   Corrections like this can be assigned through a UI or CLI. Now this ticket has an expected answer to test the next prompt iteration against.

   # Same ticket: payment cleared, but access is blocked.
   # The reviewer moves it from billing to support.
   client.feedback(
       decision_id=ticket.decision_id,
       team="support",
   )

   review queueleast certain first

   • MR

     Maya Ruiz

     Paid, but still locked out

     team

     0.54

   • JO

     Jonah Okafor

     Invoice shows the wrong plan

     team

     0.61

   • LS

     Lena Sørensen

     Export keeps timing out

     urgent

     0.66

   reviewer opens ticket #1042

   correctionticket #1042

   reviewer note

   “Payment cleared. Support needs to restore access.”

   team

   billingmodel · 0.54

   supportreviewer

   sales 

   Saved as the expected answer. The next prompt is measured against support on this ticket.
3. 03

   Prove the prompt improved

   The original prompt always sent payment questions to billing. Add the missing rule: if the customer has paid but is locked out, route to support. Replay it on the reviewed set, including this ticket.

   Now, the ticket goes to support, matching the saved correction. Compare the original and revised prompts across the same reviewed set to ensure the change helped overall before you ship it.

   group = client.group(decision.group_id)["group"]
   new_template = group["template"] + (
       "\nPaid but locked out? Route to support."
   )
   run = client.prototype(
       decision.group_id,
       template=new_template,
   )

   prompt+1 line

   1. Payment questions go to billing.
   2. +Paid but locked out? Route to support.
   3. Ticket: {ticket.text}

   client.prototype · replay on the reviewed set

   ticket #1042replayed

   MR

   Maya Ruiz

   Paid, but still locked out

   team

   originalbilling

   revisedsupport

   expectedsupport

   Matches the reviewer's correction. The revised prompt gets this ticket right without being told the answer.

   score · same reviewed set84.2%→91.0%