The AI Architect

The most expensive AI failure I deal with rarely looks like a failure. It arrives quickly. It is organized. It uses the right language. It ends with some version of: Done. Then I try to use it. A requirement is missing. A source was never checked. The agent changed something outside the scope I gave it. The answer is plausible, but there is no evidence behind it. Now I have a second job. I reopen the files. Reconstruct the context. Find the gap. Explain the correction. Rerun the work. Review it again.

Most AI harnesses can search for a hotel. Fewer can reliably compare the rooms available for your dates. The same gap appears when I ask AI to research current flight options, compare AI hardware prices, check product availability, or work through a site with filters and client-side rendering. The model may understand the task. Its normal tools still cannot see or operate the page the way a person can. I added BrowserOS as a browser layer behind my harness. It did not make the model smarter.

I stopped treating AI like chat before I had good language for why. At first it looked like a productivity upgrade. Ask a better question. Get a better answer. Paste the answer somewhere else. Repeat. That worked when the job was small. It stopped working when I wanted AI to carry real work across files, tools, decisions, checks, and follow-up fixes. The problem was not that the chatbot was dumb. The problem was that chat was the wrong interface. Chat is built around turns.

I used to lose AI coding sessions in the dumbest way. Not because the model was weak. Not because the prompt was bad. Not because I needed a longer `AGENTS.md` file. The agent would start strong, make a plan, edit the right files, run one check, hit a failure, explain the failure clearly, and then stop. Or worse, it would say the work was complete because one command passed, even though the original ask had three other requirements still unverified. That is the part nobody wants to admit.

I used to think my AI problem was prompting. Every failure looked like a prompt problem: weak context, missing examples, too vague, too clever. So I kept rewriting prompts, then watched the same task need the same manual steering the next time. The prompt got better. The work did not get repeatable. I could get one great answer from Claude, ChatGPT, or Gemini if I sat there long enough. I could nudge it, correct it, paste back missing context, and catch the hallucinated assumption.

I do not trust an agent I cannot watch fail. That became my rule after building a small Microsoft Agent Framework lab. I watched a simple agent fail for the exact reason most agent projects fail. Not because the model was bad. Not because the framework was wrong. Because I had skipped the boring part. I wanted the agent to read real data, call a model, produce a useful review, show up in a UI, and leave enough trace evidence that I could understand what happened after the fact.

I wanted more agents. That was the wrong goal. What I actually wanted was a more deterministic way to coordinate agents: clean context, focused roles, parallel work without one giant conversation turning into sludge, and a process I could inspect before trusting it. Manually launching agents worked at first. I could send one agent to review facts, one to check structure, one to test a hypothesis, and one to clean up the final draft. The context stayed cleaner than a single bloated chat.

Loading every tool all the time is the tax. I audited my agent setup last week and found a stupid tax hiding in plain sight. Before I typed a prompt, I had already spent 31% of the useful context window. Not on my code. Not on logs. Not on the task. Tool definitions. Twenty-three plugins. Eight skills. Five MCP servers. Every server politely handed the model a menu of actions, descriptions, JSON schemas, examples, and parameters. Most of those tools had nothing to do with the task in front of me.

I tested the obvious fixes. The one everyone reaches for first is not a security boundary. I had the wrong mental model for local AI agent security. I thought the problem was prompt injection. A malicious web page. A poisoned README. Some clever instruction telling the model to ignore me and leak my keys. That threat is real. But it was not the first problem. The first problem was much dumber: my agent could just read my files.

I checked my AI agent's session logs last month. 464 sessions. 1.2 billion tokens. $255.73 total. First time I had looked at them in five months of daily use. Ninety-seven percent of sessions never compacted. Subagents failed 26 percent of the time. One session had 39 consecutive tool errors. The same broken command retried 12 times with no stdin. I had no idea. The data was there. I had no visibility into any of it.