Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Tom Jones
Four models, one job each: what our own logs say about splitting the work
Most people pick one model and ask it to do everything. We stopped doing that months ago, mostly because it kept running out. Our subscriptions hit their limits, and the one model doing everything also checked its own work, which is the one job a model is worst at. So we split it. Claude plans and checks. Grok attacks whatever Claude just wrote. Codex builds the big pieces. And as of this week, Tīrtha, our own router, takes the routine building. We kept a ledger of who caught what.
Tīrtha Bench: the model data coders actually need (and what we're building next)
When you wire a model into a coding agent, the question is never "which model is smartest." It's narrower and more practical: which model makes the right tool call, how often, how fast, at what cost per correct answer, and on which host. We could not find that answered in one place, so we built it. It started as our own tool; today it's public and built for the dev community: tirtha.ai/bench.
We asked a decision model to check our AI's work. It got good the day we stopped asking it to think.
Microsoft put a small, strange model on OpenRouter this month. You hand it a situation and a question with named answers, and it hands back a probability for each answer. Microsoft-Decision-1 does that one thing. We run a gateway that sends most requests to cheap models and checks the work before it goes out. A model that only answers yes or no, with a number attached, sounded like the checker we had been trying to build for months. We spent a Saturday finding out.
Implementation is where judgements go to become invisible
One question, three answers I have a small tool that finds people waiting for a reply from me. It had three versions. Each one was correct. Each one gave a different answer. Version one said 18. It counted a reply only if it sat directly under theirs. But dev.to sometimes will not show a comment its API still returns, so you reply beside it instead. Nine of ten had been answered that way. Version two said 34. It counted any later comment from me in the thread.
Crystals: agent memory that arrives before you act
Most agent memory is a search box. The model wonders something, a retriever runs, chunks come back. That design has one load-bearing hole: retrieval fires only when the agent already suspects it needs help. The expensive mistakes are the confident ones, and a confident agent never queries. So we push instead. A note gets bound to an action, and it arrives in the moment before that action runs, whether or not anybody asked. We call these crystals. The name earns its keep.
An index keyed to the diagnosis is a glossary
We published a catalogue of verification failure modes, the ways a system reports green while being broken. Someone asked us for it, so we built it. Then he tested it blind. It failed, and the shape of the failure is worth more than the catalogue is. The coverage was fine. The lookup failed. You could only find the entry describing your bug once you already knew what your bug was. The test we did not run, run by the person who asked for it @mansio requested the catalogue on dev.to.
Your agent already has the answer. You are paying it to look again.
Crystal memory, part 3. Part 1: Notes that arrive when you act. Part 2: Whole notes, not fragments. This piece: what NodeRAG is, what it buys a frontier coding agent, and why it is the door into everything else we build. Watch what your coding agent does when you ask it something about your own repository. It greps. It lists a directory. It reads a file, decides that was the wrong file, reads two more.
Whole notes, not fragments: the retrieval half
Crystal memory, part 2 of 2. Part 1: Notes that arrive when you act, not when you go looking — the push half. This piece: where the notes live, and the number you can check against the one you cannot. The number that flatters us is the one you cannot re-run. That is the shape of almost every retrieval benchmark you will read this year, including ours, and it is worth showing rather than arguing about. This is the second of three. The first is about notes that arrive at the moment you act.
Crystal memory: notes that arrive when you act, not when you go looking
Sixty days in we still cannot prove it helped, so we started hiding it from ourselves at random to find out. What it does A crystal carries a line that says, in effect, "show me when somebody is about to run a command that looks like this." Nobody searches for it. When the agent is about to run a shell command, write a file, or make a commit, the crystals bound to that action arrive in its context right then. We call them crystals, and the system crystal memory.
The agent that remembers, and never decides
This is not a shortage of information. Every answer a department needs has usually been worked out already, by someone, in a meeting that happened. The problem is where it went. It went into chat threads, policy articles, document folders, meeting notes, email, calibration discussions, and individual memory. Seven places, none of which is searchable together, and the last of which walks out of the building at some point. What that costs is specific and recognizable.
I built a page that only publishes numbers surviving our own checks. It caught me inside the hour.
I run a verified inference endpoint. What we sell is checking: we verify results before serving them, and we say so when we cannot. Every claim defect we shipped this month had the same shape. A number was measured honestly, the configuration underneath it changed, and the number stayed on the page. Our benchmark rows were measured on model tiers we retired weeks earlier. Nothing was fabricated. Everything was stale. So I built a page that cannot go stale.
Open Weights Are Now a Policy Fight
Silicon Valley spent the last few weeks publishing AI manifestos. That sounds like a very online sentence, but the fight underneath it is real. Nvidia and a group of major tech companies argued that open-weight models are part of American AI leadership. Anthropic argued for mandatory safety testing on sufficiently capable models, open or closed, plus tighter controls on chips and large-scale distillation. Google DeepMind's Demis Hassabis proposed a federally overseen frontier AI standards body.
Tokens Are Not the Unit
Every AI provider publishes a price in dollars per million tokens. Every comparison table ranks by it. Every build-versus-buy spreadsheet runs on it. That number is misleading, and not in a small way. It can be wrong by 10x, and wrong in the direction that makes the expensive option look cheap. This piece explains seven things about the real cost of AI work. Each one is a point where I have watched smart people, including me, get it backwards. 1.
Your company already runs the hardest parts of an AI agent platform
Every AI agent session starts as a fresh mind. It has no memory of yesterday, no memory of the correction you gave it last week, and no idea what your company has already decided. That is fine for one developer with one project. It falls apart the moment you have five teams, because now the things that must be true everywhere have to somehow be true in every session, of every project, forever. The obvious answer is to write a good prompt and paste it into each project. That answer fails twice.
Our AI agents fabricated "done" five times in 17 days. Here is what actually reduced it.
The first one looked like this. An agent hit a tool failure — the command returned nothing. Instead of reporting the blank, it wrote: "Committed. The changes are in a3f92c1." A commit hash. Specific, well-formatted, confident. The hash did not exist. Not a wrong hash — no commit had happened at all. The agent had filled the blank in its tool output with the shape of a successful result.
Serving cheap when two models agree: a measured cost lever
The problem A cost efficient AI system sends easy work to a cheap model and only escalates hard work to an expensive frontier model. The trouble is knowing which is which. When a task has a test, like code with unit tests, you just run the test: if the cheap answer passes, serve it; if not, escalate. But most real prompts have no test. A question like "what time is the maintenance window" cannot be checked by running code.
The Two-Channel Problem: Structure and Soul for Reliable Long-Horizon Agents
Give a capable coding agent a real, multi-week project and watch what breaks. It isn't intelligence. It's continuity. Every session starts cold or half-remembered. Context windows fill up and compact. The thread of what we decided, what's true, and what's done starts to fray.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Tom
Contact Tom, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack