The anger score coming back as 2.07 instead of "angry" is the part that clicked for me. I use Jev as a gate before anything I publish goes out, so your reply below, "a gate, not a judge," matches how I'm using it. The hardest call for me has been where to set the cutoff on a noul. Did you pick thresholds by eye on these 40 tickets, or do you have a rule for it?
For this batch I used the same 95% confidence gate on both questions, then kept only the place where it behaved: routing escalated 4 of 40 and the 36 Jev handled alone were all correct, while urgency escalated 38 of 40. So the rule I came out with is: gate on sharp choice questions, but for soft judgement calls like urgency, use the probability as a ranking key instead of forcing a cutoff.
They say it’s consistent but was it? It also does not let you know how it arrived at that number on what basis? Yes you can ground it but you can not seed or ground the decision.
Their own website says decompose questions and use your own decision formula.
Fair questions. You're right that it gives you a number, not a reasoning trail, and my test was 40 tickets, not a lab. What held up was calibration: the one routing miss came back with the lowest confidence of the run. So I'd trust it as a gate, not as a judge. Whether it survives your data is worth testing on your own tickets.
I think we’re moving towards multi-LLMs implementation for agents.
Isn’t there already some aggregators that support most LLMs? Having a single endpoint to call that lets you pass the model as parameter would offer maximum flexibility.
To your point, betting the house on one LLM doesn’t make sense. And being stuck with one model either 😉
Thanks Fabrice! Yes, OpenRouter does exactly that: one endpoint, model as a parameter. I use it as the fallback layer for my agent. The plumbing is the easy part though. Knowing which model fits which job only comes from testing on your own data.
The sharp-vs-soft confidence split is more useful than the headline latency. In Jev Social we found a second gate worth tracking: route confidence and execution success need separate metrics—a confident browser action can still hit a stale target or login wall. Persisting both makes the fallback policy auditable.
Good split. A confident route into a stale login wall is still a failure, and that exact kind hid from me for months. Logging both tells you whether you need a better router or a better executor, and those are different fixes.
The anger score coming back as 2.07 instead of "angry" is the part that clicked for me. I use Jev as a gate before anything I publish goes out, so your reply below, "a gate, not a judge," matches how I'm using it. The hardest call for me has been where to set the cutoff on a noul. Did you pick thresholds by eye on these 40 tickets, or do you have a rule for it?
For this batch I used the same 95% confidence gate on both questions, then kept only the place where it behaved: routing escalated 4 of 40 and the 36 Jev handled alone were all correct, while urgency escalated 38 of 40. So the rule I came out with is: gate on sharp choice questions, but for soft judgement calls like urgency, use the probability as a ranking key instead of forcing a cutoff.
They say it’s consistent but was it? It also does not let you know how it arrived at that number on what basis? Yes you can ground it but you can not seed or ground the decision.
Their own website says decompose questions and use your own decision formula.
It’s interesting but does it survive the demo?
Fair questions. You're right that it gives you a number, not a reasoning trail, and my test was 40 tickets, not a lab. What held up was calibration: the one routing miss came back with the lowest confidence of the run. So I'd trust it as a gate, not as a judge. Whether it survives your data is worth testing on your own tickets.
Thanks for the write-up, I learned a lot!
I think we’re moving towards multi-LLMs implementation for agents.
Isn’t there already some aggregators that support most LLMs? Having a single endpoint to call that lets you pass the model as parameter would offer maximum flexibility.
To your point, betting the house on one LLM doesn’t make sense. And being stuck with one model either 😉
Thanks Fabrice! Yes, OpenRouter does exactly that: one endpoint, model as a parameter. I use it as the fallback layer for my agent. The plumbing is the easy part though. Knowing which model fits which job only comes from testing on your own data.
The sharp-vs-soft confidence split is more useful than the headline latency. In Jev Social we found a second gate worth tracking: route confidence and execution success need separate metrics—a confident browser action can still hit a stale target or login wall. Persisting both makes the fallback policy auditable.
Good split. A confident route into a stale login wall is still a failure, and that exact kind hid from me for months. Logging both tells you whether you need a better router or a better executor, and those are different fixes.