InferenceInference performance, quantization and cost optimization

推理(模型推理)

Using a trained model to produce output for new input.

Inference is the "run phase": the user sends a prompt and the model generates tokens. Cost is billed per token; optimization covers quantization, KV cache, batching and test-time compute.

Related terms