R3-08 Jev test plan

Supports slide 18.

What Jev is

Jev is TypeSafe’s System One model. It launched on 15 Sep 2026. It returns a choice with a probability for each candidate label, so it can score and route decisions instead of writing prose.

The test

Item	What’s on record
Where it sits	As the scoring and routing layer above our knowledge graph
What it’s for	Classification the mapping engine can’t do accurately by hand at this scale: multi-word expression tagging and entity matching
Partner	PolicyCo. Bill Butler has been using Jev since 22 Sep 2026.
Our side	Michael working with it by 25 Sep. We already call Jev in two places in our own tooling, through one gateway.
Safety	Every output passes a human Release Owner’s Gate before it’s published (R7-11).
Moves	Verified-to-paid conversion (R3-05, Q4 2026; engine v2 on the results in Q1 2027)

What the outside measurements say

A public evaluation of Jev (160 BIG-Bench Hard items, 20 Sep 2026) matched 85% of the reference labels. Its top probability bin was overconfident: a mean stated probability of 0.98 against 89% accuracy. That’s why the gate stays. A high score isn’t a reason to skip the human.

Not yet evidenced:

• a written test plan agreed with PolicyCo (pass and fail criteria, sample size, end date)
• that multi-word expressions and entity matching are the two problems that break every crosswalk (my judgement, not a measured claim)

Sources:

• Launch and PolicyCo dates: Capstone Facts Reconciled (Jev, TypeSafe AI)
• Our current Jev use and the calibration figures: Team Plugins engineering note jev-skill-fit-2026-09-28.md (citing wuyoscar/jev-skill evals/CALIBRATION_RESULTS.md)
• Release Owner’s Gate: R7-11