Haiku still wins this one

I’ve been pretty excited about Jev, SemIf, and this general idea that a lot of AI work might be better handled as specialized models, not giant all-purpose models. It has been clear for a while that the current mode of operation in AI-native development is to throw a sledgehammer at literally every problem.

I already had a good test sitting in GalexC: Gmail triage. After some deterministic rules handle the obvious stuff, Claude Haiku reads each remaining email and picks one of 16 labels.

The question was pretty simple:

What I wanted to know	Why I cared
Could Jev or SemIf do this well enough?	Email classification is narrow, repetitive, and actually useful.
Would typed choices help?	The output needs to become one label, not a paragraph.
Could I run it locally?	SemIf offered zero API spend and kept the email on my machine.
Was Haiku still worth paying for?	It already works, but “already works” isn’t a reason to stop testing.

I used real email without letting the test touch Gmail

Step	What I did
Build the corpus	Took 400 messages that already had human-applied labels.
Split by thread	Put 272 in development and saved 128 as a one-time holdout.
Blind the models	Removed the correct answer and the Gmail labels that revealed it.
Keep the inputs small	Sent only the sender, subject, and a short body preview.
Keep the test harmless	Gave none of the scorers permission to change Gmail.
Compare what would ship	Ran every answer through the same rules that decide keep, archive, flag, or review.

This was still a quick experiment. I just wanted it to be a quick experiment that could tell me something real.

Haiku stayed in production

Option	What happened	My take
Haiku	Best overall fit for the actual workflow. Fast, accurate enough, and already batched in production.	Still the winner for this one.
SemIf	Zero API spend and fully local. On the messages left after rules, it reached 28.8% exact accuracy versus Haiku’s 33.6%. It took about 14.5 minutes for 272 one-at-a-time decisions; Haiku handled its 146 residual decisions in about 50 seconds.	Very cool. Also not worth the speed and accuracy tradeoff here.
Jev	I ran the same kind of shadow test, but the evaluation agreement doesn’t permit me to publish its benchmark or performance details.	The hosted typed-choice arm didn’t replace Haiku, but I’m still very interested in the approach.

The timing workloads aren’t identical, so I wouldn’t turn those numbers into a grand speed leaderboard. They were enough to answer my question.

In sum

Finding	What I’m doing with it
Haiku remains the best operational choice for Gmail triage.	Leaving production alone.
SemIf makes cheap, private local decisions possible.	Looking for tasks where privacy or local execution matters more than latency.
Typed choices are a much better shape than generated prose for some jobs.	Looking for more tiny decisions hiding inside GalexC workflows.
The first 272 messages already answered my question.	Stopped there instead of running another 128 just to get more numbers.

That’s it. Haiku still wins this one. Jev, SemIf, and the whole category of small, specialized models are clearly very interesting.