Waldek Mastykarz

This is the seventh article in a series about Agent Experience (AX): the practice of making AI coding agents work correctly with your technology. The series covers what you can and can’t control in the agent stack, how to measure whether your extensions are helping or hurting, and how to iterate toward better outcomes. You build an eval. You run it on your machine. You get a score. Your colleague runs the same eval on their machine and gets a different score. Same scenario, same setup. What changed?

There’s advice making the rounds: replace your CLI args with a single --json payload so agents can use your tool more effectively. The thinking being, that agents already think in structured formats, and nested data maps cleanly to JSON. Flat args on the other hand, force awkward conventions like repeating --service-name to delimit multi-value groups, which is inherently ambiguous. Not to mention, that the agent needs to get the types of all values right.

A new model drops with lower per-token pricing and better benchmarks. You switch. A week later someone asks why the agent is burning 12x more tokens on the same task while producing worse output. We ran 150 agent tasks across 15 scenarios on two models, Claude Sonnet 4.6 and Claude Sonnet 5, using GitHub Copilot Chat in VS Code on Windows. The scenarios covered two types of work: architecture and design tasks grounded in Microsoft Learn documentation, and SharePoint Framework project upgrades.