Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Eric Thompson
How to Estimate the Infrastructure Cost of a Production RAG Pipeline
In our illustrative model, a production RAG pipeline costs about $1,350 a month for an internal assistant answering 100,000 questions, $6,550 for a support assistant answering 500,000, and $28,350 for enterprise search answering 2 million. That is infrastructure only, before the engineers who run it. The surprise for most teams is where the money goes. LLM generation is 78–84% of the bill in every scenario, because each answer sends several thousand tokens of retrieved text to the model.
The Cost of Self-Hosting Open-Source LLMs: GPUs, Staffing, and Operations
Self-hosting an open-source LLM in production costs roughly $150,000 to $2 million or more per year in 2026. GPUs are only part of the bill: for most companies, the engineers who run the stack cost as much as or more than the hardware. Self-hosting usually beats paying for an API only at high, steady volume or when data rules require it. • Small deployment, $150K–$300K a year: a 7B–32B model on 1–2 rented GPUs with business-hours load.
Comparing Batch and Real-Time LLM Inference Economics
Batch inference costs 50% less than real-time inference at Anthropic, OpenAI and Google, in exchange for results that can take up to 24 hours. For a workload of one million document classifications a month on a mid-tier model, that is roughly $7,000 a month in real time versus $3,500 in batch, about $42,000 a year saved (illustrative calculation below). The catch is that only work nobody is waiting on can move to batch: enrichment, classification, evaluations, reports, backfills.
Estimating the Cost of Running AI Evaluation Pipelines
Under the assumptions in this article, an AI evaluation pipeline costs about $400 a month in non-people spend for a pilot, about $5,300 a month for a single production AI feature, and about $35,000 a month for an enterprise program judging hundreds of thousands of cases. Add engineering time and the all-in figures become roughly $4,500, $25,500 and $84,000 a month with US in-house staff. Three findings hold across every scenario: • People cost more than tokens.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Eric
Contact Eric, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack