Benchmarks

All numbers come from the files in bench/data/ and are recomputed on every site build.

#lean_bench (quality)

158 generated items (72 math, 42 logic, 44 code) at three difficulty levels, seed 1, temperature 1.0, top_p 0.95, top_k 20, thinking on, one request at a time. The overall score is the mean over the three categories of the level-balanced accuracy. One seed at temperature 1.0 means roughly ±10–15 points per category.

Qwen3.8 27B Q6_K (Tersler)Ornith 1.5 35B-A3B Q4_K_M (Tersler)Qwen3.8 Flash-Next IQ3_S (moppel (RTX 5060 Ti))Qwen3.8 Flash-Next IQ3_S (Tersler)

Model and setup	Math	Logic	Code	Cut off	Time	Overall
Qwen3.8 Flash-Next IQ3_S Tersler, Strata (qr0n P100 build), RAM at 2 channels, 32k cap	100.0%	97.6%	97.7%	0	5.8 h	0.983
Qwen3.8 Flash-Next IQ3_S moppel (RTX 5060 Ti), Strata, 32k cap	98.6%	92.9%	97.7%	0	3.9 h	0.962
Qwen3.8 Flash-Next IQ3_S moppel (RTX 5060 Ti), Strata, 16k cap	98.6%	90.5%	97.7%	1	3.9 h	0.954
Ornith 1.5 35B-A3B Q4_K_M Tersler, llama.cpp, q8_0 KV, 32k cap	93.1%	88.1%	93.2%	10	3.1 h	0.916
Qwen3.8 27B Q6_K Tersler, llama.cpp, q4_0 KV, 32k cap	97.2%	90.5%	77.3%	12	4.9 h	0.878
Qwen3.8 27B Q6_K Tersler, llama.cpp, q4_0 KV, 16k cap	94.4%	88.1%	63.6%	22	3.7 h	0.821

Cut off = the answer hit the token cap before it finished; it counts as wrong. Time is wall time on one machine, so it is not comparable between Tersler and moppel.

Per task family (32k cap)

Family	Qwen3.8 27B Q6_K	Ornith 1.5 35B-A3B Q4_K_M	Qwen3.8 Flash-Next IQ3_S	Qwen3.8 Flash-Next IQ3_S
code/output	11/12	11/12	12/12	11/12
code/synthesis	23/32 (8 cut)	30/32 (1 cut)	31/32	32/32
logic/knights_knaves	6/6	6/6	6/6	6/6
logic/model_count	6/6	6/6	6/6	6/6
logic/ordering	6/6	6/6	6/6	6/6
logic/shortest_path	6/6	6/6	6/6	6/6
logic/sudoku	6/6	5/6 (1 cut)	4/6	6/6
logic/water_jugs	2/6 (4 cut)	2/6 (4 cut)	5/6	5/6
logic/zebra	6/6	6/6	6/6	6/6
math/dice	6/6	6/6	6/6	6/6
math/digit_count	6/6	6/6	5/6	6/6
math/diophantine	6/6	6/6	6/6	6/6
math/divisors	6/6	6/6	6/6	6/6
math/lattice_paths	6/6	6/6	6/6	6/6
math/lattice_polygon	4/6	5/6	6/6	6/6
math/linear_system	6/6	4/6 (2 cut)	6/6	6/6
math/modpow	6/6	6/6	6/6	6/6
math/recurrence	6/6	4/6 (2 cut)	6/6	6/6
math/urn	6/6	6/6	6/6	6/6
math/vieta	6/6	6/6	6/6	6/6
math/word_problem	6/6	6/6	6/6	6/6

#Decode speed (tersler-bench)

tersler-bench sends every prompts/*.txt to a private llama-server and appends the medians to /opt/tersler/bench/results.csv.

puzzle.txt: short reasoning task (like the reference video's puzzle test)
code.txt: code generation, mid-length output
document.txt: ~5k-token prompt generated at install time from this repo's docs (prompt-processing test; like the video's 4,000-token test)

Keep prompts unchanged between runs, or results stop being comparable. Add a new file instead of editing one.

Run (2026)	Build	Power	tg puzzle	tg doc	tg code	pp doc	Draft accept	Note
11-06 21:52	`d90ec36a3`	150 W	54.5	53.6	56.0	24	0.54	first bring-up; pp is wrong (prompt cache hit)
11-06 22:22	`d90ec36a3`	150 W	54.5	57.2	56.9	412	0.56	first bring-up on d90ec36a3
11-06 22:44	`790a3eaa1`	150 W	54.1	55.8	51.7	415	0.53	pin moved to 790a3eaa1
11-06 23:23	`83a7f0839`	150 W	54.0	53.9	54.4	434	0.55	fork head candidate, not pinned
11-07 15:51	`790a3eaa1`	150 W	58.7	59.5	57.3	406	0.60	baseline after fan control
11-07 16:06	`790a3eaa1`	125 W	51.7	53.2	48.1	361	0.56	power limit 125 W test (not adopted)
11-07 21:19	`790a3eaa1`	150 W	50.2	51.0	45.9	406	0.47	LLAMA_SPEC_SAMPLE_TEMP=0 test (reverted)

tg = decode tok/s, pp = prompt processing tok/s (~5k-token document). Model: Qwen3.8 27B UD-Q6_K with MTP speculative decoding, both cards with -sm tensor. Medians of 3 runs, 512 tokens, thinking on.

#Prefill and decode (speed test)

From scripts/speed-test.py: cold prompts (nothing cached), thinking off, up to 512 tokens, timed on the client. Prefill = prompt tokens / time to first token. Two runs per size.

Engine and setup	Prompt tokens	Prefill tok/s	Decode tok/s
Strata (qr0n P100 build), Flash-Next IQ3_S, RAM at 2 channels	44	—	55.7 / 64.5
Strata (qr0n P100 build), Flash-Next IQ3_S, RAM at 2 channels	8,446	331 / 349	41.8 / 50.0
Strata (qr0n P100 build), Flash-Next IQ3_S, RAM at 2 channels	32,001	359 / 355	45.5 / 46.8

#Multi-session tests

From scripts/concurrency-test.py; background in the research log.

Parallel requests

Server config	Parallel	Total tok/s	Per request tok/s	Draft accept
production: 1 slot, MTP, 262k context	1	38.8	51.0	0.50
4 slots, unified KV, MTP, 262k (OOM before 4 parallel)	1	37.7	49.5	0.50
4 slots, unified KV, MTP, 262k (OOM before 4 parallel)	2	42.1	25.7 · 24.6	0.45
4 slots, unified KV, no MTP, 262k	1	36.8	48.0	0.47
4 slots, unified KV, no MTP, 262k	2	40.8	25.4 · 23.6	0.42
1 slot, no MTP, 131k	1	25.7	30.5	—
1 slot, no MTP, 131k	2	27.6	30.2 · 30.6	—
4 slots, unified KV, MTP, 131k	1	35.7	46.5	0.46
4 slots, unified KV, MTP, 131k	2	41.5	24.3 · 24.7	0.43
4 slots, unified KV, MTP, 131k	4	22.8	6.5 · 6.5 · 6.8 · 6.5	0.45
4 slots, unified KV, no MTP, 131k	1	25.5	30.3	—
4 slots, unified KV, no MTP, 131k	2	40.9	23.9 · 23.9	—
4 slots, unified KV, no MTP, 131k	4	51.0	17.4 · 17.4 · 17.4 · 17.4	—
Strata (qr0n P100 build), Flash-Next IQ3_S, 131k	1	28.9	45.4	0.55
Strata (qr0n P100 build), Flash-Next IQ3_S, 131k	2	34.6	46.9 · 46.0	0.58

Swarm: root agent + 3 subagents sharing a 29k-token system prompt

prompt_n = tokens the server had to process; cached = tokens reused from the cache.

Server config	root, turn 1	subagent 1	subagent 2	subagent 3	root, turn 2
pinned build, 4 slots	76.8 s 31,541 new 0 cached	195.8 s 30 new 28,914 cached	195.9 s 28,944 new 0 cached	196.0 s 28,944 new 0 cached	13.1 s 2,653 new 28,914 cached
with server-prompt-cache-keep.patch, first version	76.9 s 31,541 new 0 cached	196.4 s 30 new 28,914 cached	196.5 s 28,944 new 0 cached	196.5 s 28,944 new 0 cached	13.3 s 2,653 new 28,914 cached
with server-prompt-cache-keep.patch (final)	78.1 s 31,541 new 0 cached	8.5 s 30 new 28,914 cached	8.5 s 30 new 28,914 cached	8.5 s 30 new 28,914 cached	16.3 s 2,653 new 28,914 cached
Strata (qr0n P100 build), Flash-Next IQ3_S, 131k	91.4 s 31,541 new 0 cached	1.5 s 30 new 28,914 cached	3.9 s 30 new 28,914 cached	2.7 s 30 new 28,914 cached	9.9 s 2,653 new 28,914 cached

Replay: prompt reuse in an agent loop

a) cold ~26k-token prompt, b) one more turn appended, c) a tool output edited in the middle, d) the same system prompt in another slot (a subagent).

Server config	a) cold	b) turn appended	c) middle edited	d) other slot, same prefix
production: 1 slot, MTP, 262k context	64.2 s 25,866 new 0 cached	5.0 s 1,171 new 25,862 cached	18.2 s 6,596 new 17,958 cached	0.9 s 20 new 17,958 cached
4 slots, unified KV, MTP, 131k	70.3 s 25,866 new 0 cached	5.0 s 1,171 new 25,862 cached	18.9 s 6,596 new 17,958 cached	46.0 s 17,978 new 0 cached
pinned build, 4 slots	66.4 s 25,866 new 0 cached	4.6 s 1,171 new 25,862 cached	18.3 s 6,596 new 17,958 cached	44.7 s 17,978 new 0 cached
with server-prompt-cache-keep.patch, first version	66.4 s 25,866 new 0 cached	4.6 s 1,171 new 25,862 cached	18.3 s 6,596 new 17,958 cached	44.8 s 17,978 new 0 cached
with server-prompt-cache-keep.patch (final)	65.9 s 25,866 new 0 cached	5.8 s 1,171 new 25,862 cached	19.7 s 6,596 new 17,958 cached	4.6 s 20 new 17,958 cached
Strata (qr0n P100 build), Flash-Next IQ3_S, 131k	75.6 s 25,880 new 0 cached	5.5 s 1,174 new 25,873 cached	19.7 s 6,596 new 17,972 cached	1.0 s 20 new 17,972 cached

#Tool calling

From scripts/tool-test.py: short conversations with tool definitions, thinking on, each case three times. The money case is a rent total with late fees (expected 801.69 EUR), with a calculator tool offered.

Case	Strata, Flash-Next IQ3_S
weather	3/3
no tool	3/3
pick tool	3/3
args	3/3
parallel	3/3
use result	3/3
error recovery	3/3
money	3/3

#Thermals (load tests)

From tersler-loadtest: sustained decode with 4096-token answers. The P100 slows down at 82 °C and shuts down at 85 °C. The fan follows the hotter card.

2026-11-06 21:56

GPU 0 °CGPU 1 °C

2026-11-07 15:53

GPU 0 °CGPU 1 °C

2026-11-07 16:08

GPU 0 °CGPU 1 °C

Run	Length	Max temp (GPU 0 / 1)	Avg power	Max fan
2026-11-06 21:56	20 min	55 / 63 °C	142 / 139 W	—
2026-11-07 15:53	12 min	66 / 75 °C	141 / 139 W	2486 rpm
2026-11-07 16:08	12 min	64 / 74 °C	120 / 117 W	2142 rpm

Benchmarks

#lean_bench (quality)

#Decode speed (tersler-bench)

#Prefill and decode (speed test)

#Multi-session tests

Parallel requests

Swarm: root agent + 3 subagents sharing a 29k-token system prompt

Replay: prompt reuse in an agent loop

#Tool calling

#Thermals (load tests)

2026-11-06 21:56

2026-11-07 15:53

2026-11-07 16:08

#Raw data