Skip to content

Latest commit

 

History

History
 
 

models

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

TT-Metalium / TT-NN Models

LLMs

Model Batch Hardware ttft (ms) t/s/u Target
t/s/u
t/s TT-Metalium Release vLLM Tenstorrent Repo Release
Qwen 3 32B (TP=8) 32 QuietBox (Wormhole) 94 22.8 30 729.6 v0.65.0-rc7 59be953
QwQ 32B (TP=8) 32 QuietBox (Wormhole) 133 25.2 30 806.4 v0.56.0-rc51 e2e0002
DeepSeek R1 Distill Llama 3.3 70B (TP=8) 32 QuietBox (Wormhole) 159 15.9 20 508.8 v0.59.0-rc53 f028da1
Llama 3.3 70B (TP=32) 32 Galaxy (Wormhole) 53 72.5 80 2268.8 v0.65.0-rc7 59be953
Llama 3.3 70B (TP=8) 32 QuietBox (Wormhole) 159 15.9 20 508.8 v0.59.0-rc53 f028da1
Llama 3.1 70B (TP=4) 32 QuietBox (Blackhole) 188* 14.9* 476.5* v0.62.0-dev20251015 1d799da
Llama 3.2 11B Vision (TP=2) 16 n300 (Wormhole) 714 18.1 17 289.6 v0.62.0-dev20251015 1d799da
Qwen 2.5 7B (TP=2) 32 n300 (Wormhole) 109 22.1 30 707.2 v0.62.0-rc35 ced0161
Qwen 2.5 72B (TP=8) 32 QuietBox (Wormhole) 223 15.4 20 492.8 v0.62.0-rc25 e7c329b
Falcon 7B 32 n150 (Wormhole) 70 18.5 26 592.0 v0.62.0-dev20251015
Falcon 7B (DP=8) 256 QuietBox (Wormhole) 87 15.9 26 4070.4 v0.62.0-dev20250926
Falcon 7B (DP=32) 1024 Galaxy (Wormhole) 121 13.2 26 13516.8 v0.62.0-rc25
Falcon 40B (TP=8) 32 QuietBox (Wormhole) 11.9 36 380.8 v0.59.0-rc38
Llama 3.1 8B 32 p100 (Blackhole) 61* 29.0* 928.0* v0.64.0-dev20251030 0edd242
Llama 3.1 8B 32 p150 (Blackhole) 57* 33.1* 1059.2* v0.64.0-dev20251030 0edd242
Llama 3.1 8B (DP=2) 64 2 x p150 (Blackhole) 64* 18.6* 1190.4* v0.59.0-rc3 739dcaa
Llama 3.1 8B 32 n150 (Wormhole) 104 26.0 23 832.0 v0.62.0-rc35 ced0161
Llama 3.2 1B 32 n150 (Wormhole) 23 80.5 160 2576.0 v0.62.0-dev20251015 1d799da
Llama 3.2 3B 32 n150 (Wormhole) 52 46.6 60 1491.2 v0.62.0-rc35 ced0161
Mamba 2.8B 32 n150 (Wormhole) 35 14.1 41 451.2 v0.59.0-rc38
Mistral 7B 32 n150 (Wormhole) 99 28.7 23 918.4 v0.64.0-dev20251030 0edd242
Mixtral 8x7B (TP=8) 32 QuietBox (Wormhole) 122 24.9 33 796.8 v0.62.0-dev20251015

Last Update: Dec 8, 2025

Notes:

  • ttft = time to first token | t/s/u = tokens/second/user | t/s = tokens/second; where t/s = t/s/u * batch.
  • TP = Tensor Parallel, DP = Data Parallel; Defines parallelization factors across multiple devices.
  • The reported LLM performance is for an input sequence length (number of rows filled in the KV cache) of 128 for all models except Mamba (which can accept any sequence length).
  • The t/s/u reported is the throughput of the first token generated after prefill, i.e. 1 / inter token latency.
  • Performance numbers were collected using the tt-metal model demos (accessible via the model links). If running with a vLLM inference server, performance may be different.
  • * Blackhole software optimization is under active development. Please join us in shaping the future of open source AI!
    [Discord] [Developer Hub]
  • For more information regarding vLLM installation and environment creation visit the Tenstorrent vLLM TT plugin README.

Speech-to-Text

Model Batch Hardware ttft (ms) t/s/u Target t/s/u t/s TT-Metalium Release
Whisper (distil-large-v3) 1 n150 (Wormhole) 163 105.0 45 105.0 v0.65.0-dev20251208
Whisper (distil-large-v3) 1 p150 (Blackhole) 63 263.4 263.4 v0.65.0-dev20251208

Diffusion Models

Model Batch Hardware Sec/Image Target Sec/Image Release
Stable Diffusion 1.4 (512x512) 1 n150 (Wormhole) 4.83 3
Stable Diffusion 1.4 (512x512) 1 p150 (Blackhole) 3.15
Stable Diffusion 3.5 Medium (512x512) 1 n150 (Wormhole) 16 10

Notes:

  • Stable Diffusion sec/image is based on the time elapsed from submitting the input prompt to receiving the image from the VAE decoder.

CNNs and Vision Transformers

Classification models

Model Batch Hardware Image/sec Target Image/sec Release
ResNet-50 (224x224) 16 n150 (Wormhole) 4,700 7,000 v0.59.0
ResNet-50 (224x224) (DP=2) 32 n300 (Wormhole) 9,200 14,000 v0.59.0
ResNet-50 (224x224) 16 p150 (Blackhole) 12,600 14,000 v0.59.0
ResNet-50 (224x224) (DP=8) 128 QuietBox (Wormhole) 35,800 56,000 v0.59.0
ResNet-50 (224x224) (DP=32) 512 Galaxy (Wormhole) 96,800 224,000 v0.59.0
ViT-base (224x224) 8 n150 (Wormhole) 1,370 1,600 v0.60.0-rc4
ViT-base (224x224) (DP=2) 16 n300 (Wormhole) 1,900 3,200 v0.60.0-rc4
ViT-base (224x224) 10 p150 (Blackhole) 3,700 3,200
ViT-base (224x224) (DP=8) 64 QuietBox (Wormhole) 7,700 12,800 v0.60.0-rc4
MobileNet-v2 (224x224) 10 n150 (Wormhole) 3,030 3,500
MobileNet-v2 (224x224) (DP=2) 20 n300 (Wormhole) 5,280 7,000

Segmentation

Model Batch Hardware Frame/sec (FPS) Target FPS Release
UNet - VGG19 (256x256) 1 n150 (Wormhole) 198 150
UNet - VGG19 (256x256) (DP=2) 2 n300 (Wormhole) 318 300
UNet - VGG19 (256x256) 1 p150 (Blackhole) 320 300
UNet - Vanilla (480x640) 1 n150 (Wormhole) 60 240
UNet - Vanilla (480x640) 2 n300 (Wormhole) 119 480
SegFormer Semantic Segmentation (512x512) 1 n150 (Wormhole) 132 360
SegFormer Semantic Segmentation (512x512) (DP=2) 2 n300 (Wormhole) 248 720
UFLD - v2 (320x800) 1 n150 (Wormhole) 365 600
UFLD - v2 (320x800) (DP=2) 2 n300 (Wormhole) 637 1200
UFLD - v2 (320x800) 1 p150 (Blackhole) 551 1200

NLPs

Model Batch Hardware Sentence/sec Target sentence/sec Release
BERT-Large 8 n150 (Wormhole) 270 400
Sentence-Bert (backbone: bert-base) 8 n150 (Wormhole) 433 550
Sentence-Bert (backbone: bert-base) (DP=2) 16 n300 (Wormhole) 772 1100
Sentence-Bert (backbone: bert-base) 8 p150 (Blackhole) 929 1100
ModernBERT-base 8 n300 (Wormhole) 365

Model Demos

Latest Releases

Release Release Date
0.65.0 ETA Dec 2025
0.64.0 Oct 29, 2025
0.63.0 Sep 22, 2025
0.62.2 Aug 20, 2025
0.61.0 Skipped
0.60.0 Jul 16, 2025
0.59.0 Jun 18, 2025
0.58.0 May 13, 2025
0.57.0 Apr 15, 2025
0.56.0 Mar 7, 2025

Visit the releases folder for details on releases, release notes, and estimated release dates.

Release methodology

  • Models that are advertised as part of release, usually the demo models, are treated as first-class citizens, and therefore are treated as tests.
  • Model writers are responsible for ensuring their demo model tests are always passing. Any failure is treated highest-priority (or P0) failure.
  • Model writers are responsible for advertising which release tag (including release candidates) contains passing tests for their demo models.
  • Model writers are responsible for updating their perf metrics for the demo models at a regular cadence. Currently, the cadence is at least every 2 weeks.