You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ttft = time to first token | t/s/u = tokens/second/user | t/s = tokens/second; where t/s = t/s/u * batch.
TP = Tensor Parallel, DP = Data Parallel; Defines parallelization factors across multiple devices.
The reported LLM performance is for an input sequence length (number of rows filled in the KV cache) of 128 for all models except Mamba (which can accept any sequence length).
The t/s/u reported is the throughput of the first token generated after prefill, i.e. 1 / inter token latency.
Performance numbers were collected using the tt-metal model demos (accessible via the model links). If running with a vLLM inference server, performance may be different.
* Blackhole software optimization is under active development. Please join us in shaping the future of open source AI! [Discord][Developer Hub]
Visit the releases folder for details on releases, release notes, and estimated release dates.
Release methodology
Models that are advertised as part of release, usually the demo models, are treated as first-class citizens, and therefore are treated as tests.
Model writers are responsible for ensuring their demo model tests are always passing. Any failure is treated highest-priority (or P0) failure.
Model writers are responsible for advertising which release tag (including release candidates) contains passing tests for their demo models.
Model writers are responsible for updating their perf metrics for the demo models at a regular cadence. Currently, the cadence is at least every 2 weeks.