Sungsoo Ha

Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. However, most existing implementations focus on generating a single sequence. Real-world generative-AI applications often require multiple responses, and how to perform speculative decoding in a batched setting while preserving its latency benefits poses non-trivial challenges.