Sid Shanker

We heard from customers coming from other inference platforms that while updating their models, they were stuck choosing between blue-green deployments and hard cutovers. Blue-green deployments require a full parallel fleet alongside the existing one, effectively doubling compute spend for the duration of the rollout (and challenging to do during a capacity crunch). Hard cutovers are cheaper but all-or-nothing: if you run into issues, there's no pausing mid-rollout.

Developers want the ability to test production workflows and workloads without the consequences of failing in production.

We’re excited to launch canary deployments, enabling you to gradually shift traffic to new model deployments over custom time windows! Rolling out new deployments is a fraught process, especially when it comes to complex generative AI models. From the model quality to performing at scale and interacting with other moving parts in your pipeline, there’s always a non-negligible risk that something could go wrong.