Pool routing

Every model is served by one or more pools. Which pool a request lands in depends on your pool preferences, the pool the request asks for explicitly, and how much capacity each pool has left.

We never silently route your requests to other models.

No preference

• The model’s default pool is attempted first.
• If it’s out of capacity, the request is re-routed to the next cheaper pool with capacity.

With a preference

Preferences are set on the pools settings page.

• Your preferred pool is attempted (per-model first, then the model’s family, then “all models”).
• If that pool can’t serve the model, the “no preference” path applies.
• If it’s out of capacity and “allow cheaper” is enabled, the request is re-routed to a cheaper pool (never to a different model).
• Otherwise, the request fails.

With an explicit pool

A request picks a pool explicitly with a pool: model prefix or the x-knt-pool header.

• Only that pool is attempted, preferences are ignored.
• If it’s out of capacity, the request fails - unless the request sends x-knt-allow-cheaper: true, which re-routes it to a cheaper pool.

With a meta pool

cheapest and fastest are meta pools: they resolve to a ranking rather than a single pool.

• cheapest ranks the pools pricing the model, cheapest first
• fastest ranks them by measured time-to-first-token and tokens per second.
  • uses the model’s default pool where there is not enough traffic to measure.