🔬 ParisiPlace GPU Inference Benchmark

Token generation speed — how many tokens (≈¾ of a word) the model outputs per second during a conversation. This is the number you feel as a user. At 80 tok/s, a 400-word response streams in ~5 seconds. At 30 tok/s, it takes ~13 seconds.

tg128 = generating 128 tokens. tg512 = generating 512 tokens. For a well-optimised dense model, these should be nearly identical — the GPU reads weights at constant bandwidth regardless of output length.

Prompt processing speed — how fast the model reads your input and the entire conversation history before generating the first word. pp512 = processing 512 tokens at once, pp4096 = 4096 tokens.

At the start of every response, the model must process the full context window (up to 131,000 tokens for our Librarian). Higher pp = faster time-to-first-token. Especially important for long documents or large conversation histories.

Context scaling test. Dense models (Qwen, Ornith) often get faster at larger batch sizes — the GPU's 5120 CUDA cores are better utilised when processing more tokens simultaneously.

MoE models (Gemma4-26B with ~4B active parameters out of 25B) show a drop at pp4096 — the sparse expert routing doesn't parallelise as efficiently at large batches.

GPU memory used during the benchmark. Whatever VRAM the model doesn't occupy becomes available for the KV cache — the memory buffer storing the conversation history.

More VRAM headroom = longer conversations before slowdown. The current Librarian (13.4 GiB) leaves only ~2.7 GiB for KV cache. The 9B candidates (6.9 GiB) leave ~9 GiB — enabling much longer conversations at full speed.

GPU wattage during inference. The V100 SXM2 has a 300W TDP; the Titan V is 250W. High power during tg confirms the GPU is compute-saturated — fully utilised, which is ideal.

Very low power during tg (e.g. the Qwen3.8-27B IQ4_XS at ~39W) indicates CPU offload is occurring — most layers are on RAM, not GPU, and the GPU is nearly idle while waiting for data.

GPU core temperature at peak load. The V100 SXM2 HBM2 memory has an 85°C thermal limit — our benches peaked at 64–66°C, giving 19–21°C of thermal headroom.

Running 24/7 as the Librarian under sustained load will be warmer than a short benchmark. If temperatures approach 80°C consistently, better airflow or reduced clock speeds may be needed. No throttling was observed in any of our runs.

Shader processor clock speed during inference. The V100 SXM2 boosts to 1530 MHz under full load; the Titan V reaches 1335 MHz.

A consistently high SM clock during tg confirms no thermal or power throttling. If the clock drops significantly mid-bench, the GPU is protecting itself from overheating — a signal that cooling needs attention.

HBM2 bandwidth utilisation — the fraction of the GPU's memory bus being used. LLM token generation is memory-bandwidth-bound: for each output token, the GPU must read the entire set of model weights from HBM2 memory.

80–99% memory utilisation during tg is expected and healthy — it means the GPU is spending its time moving weight data as fast as the HBM2 bus allows. Low utilisation suggests a bottleneck elsewhere (e.g. CPU-side data starvation).

Wall-clock time for the complete benchmark of one model — including Docker container startup (V100) or process launch (Titan V), model weight loading into VRAM, and all test repetitions.

The 9B models take 47–105s per run; the 26B baseline takes 124–152s (larger model = longer load). Docker adds ~35–40s overhead vs the native binary on Calcifer — which explains why the Titan V bench (3m55s) was faster than the V100 bench (8m48s) despite the V100 being the faster GPU.

Measurement stability across 3 repetitions. A low ± (e.g. ±0.30) means the GPU is running at a stable, predictable speed — good for a 24/7 production workload.

A high ± (e.g. Gemma4-26B tg512: ±15.78) means significant run-to-run variation — in this case caused by the MoE model's non-deterministic expert routing at longer sequences. Not shown in bar charts but visible in the raw bench logs.