吞吐量

The number of requests or tokens a model handles per unit of time.

Throughput determines serving cost and capacity: higher throughput means lower cost per token. Batching multiple requests is the core lever, at the expense of latency.