AI
GAUSS models LLM serving latency beyond a single throughput number
GAUSS is a workload-aware simulator for planning large-language-model inference. It models prefill and decode separately, estimates time to first token, time between tokens, sustainable request rates, and latency tails, and uses calibrated latency surfaces with Monte Carlo sampling. The authors say it can evaluate workloads, routing policies, hardware, and scheduling choices that have not yet been deployed. Its central operational claim is that aggregate throughput can hide user-visible tail degradation, making fuller workload and latency modeling important for capacity decisions.
Editorial summary · Publisher article
What matters
- The simulator models both time to first token and time between tokens.
- It uses a queueing model and ensemble Monte Carlo sampling instead of replaying one trace.
- The framework is intended for operating-point selection, traffic segmentation, and scheduling analysis.
Context & caveats
- The supplied material is an abstract and does not provide comparative validation results or implementation details.
Why this made the edition
The paper presents a concrete simulator design for LLM inference operations, including phase-specific latency modeling, queueing, Monte Carlo estimation, and several practical capacity-planning uses. The supplied abstract supports a useful technical report, although validation details are not included.
Source material