#Research log

How the Tersler hardware was chosen. Prices are eBay.de listings as seen on 2026-10-31; they change daily, so treat them as a snapshot. The final parts list is in bom.md.

#Goal and constraints

• Host two NVIDIA Tesla P100 PCIe 16 GB cards (already owned) for llama.cpp inference, models mostly fitting in the 32 GB VRAM.
• Late 2026 market: GPU and RAM prices are high because of supply shortages, and that includes DDR4.
• Budget: ≤ €250 for everything except the GPUs (adapters and PSU included). No case needed; GPU cooling is DIY.
• Owner has an engineering/CS background, so DIY power and cooling are fine.
• Hard constraint added later: no 40 mm fans (too loud).

#What a P100 needs from a board

• Above 4G decoding. Each card needs a 16 GB BAR1 mapping. Older BIOSes cannot do it, which is the usual "black screen" problem.
• Two x16-size slots with dual-slot spacing. x8 electrical is fine for layer split (the reference video measured ~2% loss with one card at x8, using tensor split).
• A display path. The P100 has no outputs, so the board needs onboard VGA/BMC, an iGPU, or a cheap extra card.
• Power: up to 250 W per card, through an 8-pin CPU/EPS connector, not PCIe 8-pin. In practice the cards run at 150 W (see the reference recipe).
• Passive cards: they need ducted forced airflow.
• Software: NVIDIA's R580 driver branch is the last one with Pascal support; CUDA 13 dropped Pascal, so stay on CUDA 12.x.

#Platform options considered (first pass)

Key market observation: used DDR3 server RAM is almost free (4 × 8 GB = 32 GB for €12.50–40). Single 16 GB DDR4 registered ECC sticks cost €45–60 used. The only DDR4 32 GB-class price seen was €165 for a 16 GB stick from a premium retailer.

#eBay.de candidates seen (2026-10-31)

#Rejected as too expensive

• Supermicro H11SSL-i: board alone ~€449–529 (from China); with EPYC 7282 ~€537–556. Rev 2.0 needed for EPYC 7002.
• Gigabyte MW51-HP0: ~€397 + €80 shipping (China).
• HP Z840 complete: €659 (2× E5-2620 v3, 32 GB) to €795 (2× E5-2680 v4, 64 GB).
• Dell T5810 complete: from ~€300 (16 GB). Its 685 W PSU is tight for 2 × P100 without a second supply.

#≤ €250 board candidates

#Why the ASUS Z10PA-U8

Verified from the ASUS user guide (stored in reference/) and the seller's photo:

• Slots: PCIE1 x8 (x4 Gen2, PCH), PCI2, PCIE3 x8, PCIE4 x16, PCIE5 x8, PCIE6 x16, MIO1. PCIE4/PCIE6 drop to x8 only if PCIE3/PCIE5 are used. Cards in PCIE6 + PCIE4 cover PCIE5/PCIE3, so they run x16/x16 Gen3, both on CPU1. That suits P2P, and the adjacent cards let one fan cool both.
  • The seller's text ("PCI-E x8, x16, PCI") was wrong; the photo and the manual agree.
• BIOS: "Above 4G Decoding" under Advanced → PCI Subsystem Settings (manual §4.4.8), default Disabled. VT-d is enabled by default (§4.5.5).
• Video: onboard ASPEED AST2400 VGA.
• CPU socket: square ILM, ~80 × 80 mm, measured on the seller's photo against the 305 × 244 mm board outline. A first measurement wrongly suggested a narrow ILM; it had picked up the ILM's own hold-down screws.
• Power connectors: 24-pin + 8-pin CPU (EPS).
• M.2 (NGFF1): 2242 SATA only, shares bandwidth with a SATA port, so it is no faster than a 2.5" SATA SSD. A faster option is NVMe on a PCIe adapter in PCIE1 (Gen2 x4, ~2 GB/s); NVMe boot support is unverified.
• CPU support: the guide lists E5-2600/1600 v3. v4 needs a newer BIOS, and flashing requires a working v3 CPU. So the build uses a v3 chip.

#Power decisions

• Peak estimate at 150 W per card: CPU 135 + board/RAM/SSD ~50 + 2 × 150 ≈ 485 W.
• Options compared:
  • Server PSU + breakout board. Rejected by the owner (no mining/server PSUs).
  • Two ATX PSUs. Saved only ~€6.50 at 250 W per card.
  • One FSP 1000 W (needs a C19 cable).
  • One Seasonic SSP-650RT 650 W Gold: tested, 12-month warranty, and a 4+4 CPU cable plus a separate 8-pin EPS. Chosen; about 165–175 W headroom at 150 W per card.
• Cable plan: board ← 4+4 CPU; P100 #1 ← native 8-pin EPS; P100 #2 ← homemade 2 × PCIe 6+2 → EPS adapter (one Goobay EPS extension). See hardware-notes.md. In the end not needed: the cards came with Y-adapters, and P100 #2 runs on one of those.
• An earlier pick (HP 1200 W + breakout board) turned out to be an auction, not Buy-It-Now. After that, every listing was re-checked by its bid/buy button.

#CPU: E5-2690 v3 vs v4

The reference video used a v4 (14C, 2.6/3.5 GHz, 135 W, DDR4-2400). The v3 (12C, 2.6/3.5 GHz, 135 W, DDR4-2133) has the same boost clock, and Broadwell has only ~5% better IPC. Inference is GPU-bound, so the expected difference is ≤ ~1% (estimate, not measured). The v4 costs ~€13 more plus the BIOS risk, so the v3 was chosen.

#Comparison with the reference build (Dell T5810)

From Dell's spec sheet (reference/):

• C612, LGA2011-3; 2 × PCIe x16 Gen3.
• PSU 425/685/825 W; Dell allows 500 W of graphics only with the 825 W unit.
• Quad-channel DDR4 RDIMM, up to 256 GB.

Same platform class as our build. Their 80 GB RAM only matters for page cache and for compiling with many parallel jobs (models stay in VRAM).

#Open items found while reviewing the video recipe

• The fork uses -sm tensor, not layer split, so PCIe width and P2P matter more than first assumed. x16/x16 covers it.
• P2P (GGML_CUDA_P2P=1) needs the kernel parameter iommu=pt.
• Build for sm_60 only; a second architecture disables the Pascal tuning.
• -DGGML_CUDA_FA_QUANTS=all is a heavy compile; limit parallel jobs on 24 GB RAM.

#Alternative engine: Kindi (checked 2026-11-05)

ahtx/kindi v0.4.0 (AGPL-3.0, one main author, ~120 stars) added the Tesla P100 and claims "1.9 to 2.7 times as fast as llama.cpp". What that number means:

• Different models. Kindi runs only Prism ML's Ternary-Bonsai-2 models (ternary weights, ~1.6–1.75 bit), Nemotron 3.5 Bolt E4B and its own AH-Nanbeige4.3-33B IQ3_XXS. It can't run our Qwen3.8-27B Q6_K.
• Weak baseline. The comparison is against Prism's llama.cpp fork (c96baf4) on the same ternary file, not against the Kmic P100 fork.
• No speculation on the P100. Kindi turns it off on sm_60 (it says a verify pass costs as much as the single steps it replaces). Our recipe gets much of its speed from MTP.

Where Kindi might still be interesting:

• The full 262K context of the 27B fits on two cards (8.6 GB free each); prompt read at that depth is slow (67.6 tok/s).
• A prompt cache that resumes the recurrent (DeltaNet) state, reportedly much faster than llama.cpp's on repeated long prompts (agent use).
• AH-Nanbeige4.3-33B on two P100s: 21.0 tok/s (llama.cpp 9.6).
• Docker image with CUDA 10.2; only needs the host driver, so it can sit next to our install.

Verdict: not a replacement. Quality of a ternary 27B vs Q6_K is unmeasured here; worth one evening on the rig after the main bring-up, using the same bench prompts. Watched in install/upstreams.list.

#Alternative model: Nanbeige4.3-33B (checked 2026-11-05)

• The claim: The vendor card (Nanbeige/Nanbeige4.3-33B) claims it beats Qwen3.8-27B on most agentic-coding benchmarks. Examples: Terminal-Bench 2.1 80.9 vs 73.0, and DeepSWE 65.5 vs 42.5. Qwen3.8 leads on SWE-Bench Pro, 61.7 vs 60.3. These are vendor numbers for the full BF16 model at its highest thinking effort. The AH IQ3_XXS card compares only against Kindi's ternary Bonsai 27B, and it reports a large quantization loss: 82.8 % same top token, KL 0.225.
• Architecture:
  • Plain LlamaForCausalLM, dense 33.1B.
  • 52 layers, hidden size 7168, 56 heads / 8 KV heads, head_dim 128.
  • Vocabulary 166,912 tokens; context up to 512K.
  • No MTP head.
• Support in the pinned fork (d90ec36a3):
  • LLM_ARCH_LLAMA is supported.
  • -sm tensor is allowed (Llama is not on the fork's exclusion list).
  • IQ3_XXS runs on the generic path with the fork's emulated DP4A for sm_60. The P100-tuned matvec paths cover Q4_0, Q8_0, Q4_K and Q5_K.
• What changes vs our config:
  • Drop the MTP / draft flags.
  • Use a K-quant. Q4_K_M is 20.8 GB, Q5_K_S 23.7 GB (bartowski).
  • Use a smaller context: the dense GQA KV cache is ~110 KB/token at q8_0 and ~60 KB/token at q4_0, so 128K at q4_0 ≈ 7.9 GB.
  • Example override: install/tersler.local.conf.example.
• Speed (unmeasured):
  • Without speculation, expect well below the Qwen MTP numbers. Each card streams ~10 GB per token at Q4_K_M, so ~50 tok/s is the bandwidth ceiling; realistically maybe 20–30.
  • Kindi measured 9.6 tok/s for llama.cpp (Prism fork) with the IQ3_XXS file on 2× P100.
• Later: a draft model could bring speculation back. Nanbeige announced a distilled 3B, and a DFlash draft exists (ProCreations/Nanbeige4.3-33B-DFlash2); llama.cpp support for it is unknown.

#Alternative engine: Strata (checked 2026-11-06)

Niko1221/Strata, MIT. An inference engine plus desktop app built around Qwen3.8-Flash-Next (MoE, ~3B active). It keeps the hot experts in VRAM and the rest in system RAM (expert cache). Very active: 41k stars, releases every 2-3 days (v0.2.11 on 2026-11-05), >1000 open issues.

• Pascal/P100: works, but it is community-tested, not officially supported.
  • The official engine packages carry no sm_60 code; build from source with CUDA 12.9.
  • Start with strata serve --profile, because strata launch can replace a self-built engine with the downloaded one (open question #5114).
  • Unfixed sm_60 correctness bug: the DP4A fallback (include/strata/kernels/dp4a.hpp) declares its accumulator "+r", so NVCC may alias it with an input. A forced probe gives ~62k wrong dot products per 65,536 on a P100. The fix is PR #4274 ("+&r", three lines), still unmerged in v0.2.11 and on main. Not proven to hit real kernels, but it is a compiler-dependent wrong-result path on exactly our cards: apply it to any sm_60 build.
• Quality reports, not P100-specific:
  • #2685 (closed in a tracker reset, not fixed): Strata answered worse than llama.cpp with the same IQ3_S file. Causes were greedy sampling by default (set temperature 0.6, top_p 0.95, top_k 20), tool-call recovery off, and nondeterminism in the prompt path (STRATA_PREFILL_MMQ=0 / STRATA_PREFILL_CPU_SHARE=0 made runs reproducible on some cards).
  • #4059 (open): identical temperature-0 requests give different logprobs.
  • Two-card layer split vs reference: first-token KL 0.0316, 92.3% top-1 (v0.2.11 release notes).
• Measured on P100s:
  • 2x P100 write-up: Q2_0 (66 GB), 31 GB RAM, PCIe x4. 59.8 tok/s on code at 128K context; prose 37-39. KV int8; 256K costs ~12%, because the KV takes expert-cache VRAM. No quality measurement at all.
  • #5125: three P100s in a six-card split, IQ3_S, with the #4274 fix. 56.2 tok/s decode (code 58.7, prose 45.2); prompt reading 391 / 1010 / 1490 tok/s at 4K / 32K / 128K. 793 requests, no errors.
• For this rig:
  • Needs ≥32 GB RAM (64 GB after the upgrade) and a 46-66 GB model.
  • Decode about the same as our 27B (55-60 tok/s on code, slower on prose). Much faster prefill at long context.
  • Context is not free: KV and the expert cache share the VRAM, so more context means fewer resident experts and slower decode.
  • Worth trying after the RAM upgrade, as a second service, built from source with #4274. Gate it on a quality check (same prompts vs llama.cpp with the same GGUF) before trusting it.
• Same file in llama.cpp (checked 2026-11-06): possible.
  • Upstream llama.cpp keeps MoE experts in host RAM and caches the hot ones on the GPUs: --cpu-moe + --moe-cache-mib N (#29887, 2026-10-07; several GPUs by layer #30112; SLRU #30637).
  • The fork head 83a7f0839 has it; our pin 790a3eaa1 has only --cpu-moe, so the experts run on the CPU there.
  • The MoE cache refuses -sm tensor ("does not support tensor parallelism"), so use -sm layer.
  • The Flash-Next architecture (qwen4exp) is supported in both builds.
  • ISTA-DASLab IQ3_S = 54.8 GB shard 1 + 28.8 GB shard 2. Shard 2 holds a single tensor, per_layer_token_embd: a lookup table the model only reads for the tokens in use. Via mmap it can stay mostly on the SSD.
  • #2685 compared Strata and llama.cpp exactly this way (--cpu-moe --moe-cache-mib 2048 --load-mode none, 12 GB GPU, 60 GB RAM).

#Strata on Tersler: prepared (2026-11-10)

• Engine: qr0n/p100-strata 8b1b0f5: upstream v0.2.2 plus 12 P100 patches (the build the Junk GPU Lab video used). Plus our install/patches/strata-sm60-dp4a-alias.patch (#4274, still unmerged in v0.2.13; the bug is present in qr0n's tree). Newer upstream (v0.2.13) has none of the P100 decode patches.
• Model: our ISTA-DASLab IQ3_S (83.6 GB). qr0n and junkgpulab used IQ3_XXS (75.8 GB); patch 0007 covers IQ3_S. The weight pack keeps the experts in the GGUF (no --experts-bin), so it costs little disk space.
• Config install/strata/flash-next-p100.json (since 2026-11-10 also 262k context and a CPU image encoder): qr0n's peer layout (card 1 as expert tier), int8 KV, MTP spec 4, STRATA_ROUTE_TAIL_SKIP=0 and qr0n's fp16 switches. Changed: --ple-io direct (the 28.8 GB n-gram table read from the SSD; with 62 GB RAM it can't sit in RAM next to the expert arena) and 131k context.
• Run: scripts/strata-test.sh prep | start | stop | status, port 8082. start stops tersler-llama.
• Caveat: RAM bandwidth is at two channels' worth until channel D works (see hardware notes), and Strata computes cache misses on the CPU. Results now are a baseline; repeat after the CPU swap.
• Test plan: speed (short prompt, 7k and 27k documents: decode and prefill), then lean_bench with the same settings as the moppel run (0.962) as the quality gate.

• First speed test (2026-11-10, RAM at two channels' worth, scripts/speed-test.py, thinking off, 512 tokens, cold prompts, client clock):

  Prompt	Prefill tok/s	Decode tok/s
  short (44 tokens)	—	55.7 / 64.5
  8.4k-token document	331 / 349	41.8 / 50.0
  32k-token document	359 / 355	45.5 / 46.8

  • Load: 3.5 min from the SSD (experts at 0.26 GiB/s). 11,938 of 24,576 experts on the GPUs (4,822 on card 0, 7,116 on the peer card 1), the same as qr0n's 11,950. Decode hit rate 79-94 %. RAM: 52 of 62 GB used.
  • About the video's numbers (55 tok/s, 382 tok/s prefill at 40k), with a larger quant and half the RAM bandwidth. Compared with our 27B on llama.cpp: same decode, slower prefill (~410 tok/s at 5k).
  • Cards stayed cool (max 73 °C, ~35-55 W): they mostly wait for the CPU and PCIe.

#lean_bench: Tersler vs Strata (2026-11-07)

~/src/llm-benchmark (lean_bench), seed 1, size standard (158 items). Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0; thinking on; one request at a time.

• 27B rerun with the 32k cap: of its 22 truncated items, 9 became correct and 1 finished wrong. The other 12 still hit 32k: all 4 water-jug puzzles and 8 code tasks, some of them L1. Its average synthesis answer was 15.7k tokens, against 10.0k for Flash-Next.
• Strata on Tersler (2026-11-10): qr0n's P100 build, IQ3_S, int8 KV, MTP, RAM at two channels. 156/158; misses: logic/water_jugs/L3/1 (answered 7, correct is 12) and code/output/L3/3. Nothing truncated; the 27B never finished a level-2/3 water-jug puzzle, Strata solved 5 of 6. ~48 tok/s, 5.8 h for the run (code-writing answers average 15.5k tokens).
• Agent tests on the Strata service (2026-11-10, 131k): tool calling 24/24 (scripts/tool-test.py: tool choice, arguments, parallel calls, tool results, error recovery, rent total with late fees 801.69 three times). Replay/swarm (scripts/concurrency-test.py): shared prefixes are reused without any patch: subagent with the root's 29k system prompt 1.0-3.9 s (llama.cpp stock: 45-196 s, with our patch 4.6-8.5 s); continued and edited conversations as fast as llama.cpp. Two parallel chats: 46-47 tok/s each.
• Long-context agent session on the Strata service (2026-11-10, 262k, scripts/longctx-test.py): tool outputs from llama.cpp's sources with three planted facts at ~10/50/90 % depth. All three recalled exactly.
  • Cold 122.8k tokens: 368 s (334 tok/s). Extending to 203.9k (81k new, cache reused): 283 s (287 tok/s).
  • Follow-up question at 204k: 0.7 s to the first token (all cached). Decode at 204k: 37.5 tok/s (about 47 at short context).
  • RAM 56 of 62 GB in use; GPU 0 at 73 °C.
• Ornith 1.5 (2026-11-10): q8_0 KV, -c 65536, MTP, ~97 tok/s on long answers; 158 items in 3.1 h. All 10 misses from truncation are at 32k: 4 water-jug, 2 recurrence, 2 linear_system, 1 sudoku, 1 synthesis. Code is far better than the 27B (93 vs 77 %); math and logic are about the same.
• Flash-Next: only 1 item truncated at 16k, and it was correct at 32k.
• Limits: one seed, sampled at temperature 1.0, so ±10-15 points per category (Wilson CI). Math and logic overlap; code synthesis does not.

#Junk GPU Lab videos (checked 2026-11-07)

Channel: https://www.youtube.com/@junkgpulab (5 videos; newest seen: 0aBLGbfzvUs, checked 2026-11-10).

• JvLUoJYUjDM "Qwen 3.8 27B at 67 tok/s": the reference build (recipe).
• oYLZFgduLDU "Can Two Old Tesla P100s Replace My $200 Claude?" (recipe 2.0):
  • LLAMA_SPEC_SAMPLE_TEMP=0, not 1.0: +9 % prose, +4 % code on his fork. On our Kmic build 0 is 12-20 % slower (draft accept 0.6 → 0.47; see changelog), so we keep 1.0.
  • For real tasks he uses -ctk q8_0 -ctv q8_0, because q4_0 cut long thinking short. That may matter for our lean_bench truncations. Costs about twice the KV memory.
  • His fork: https://github.com/junkgpulab/llama.cpp-p100 (current upstream + Kmic's P100 work + a fix for MTP printing "4444" after the first question).
  • 27B Q6: 6/8 on his answer-keyed tasks; Flash-Next: 8/8.
• zT09gR54e9w "Watch This Before You Use Strata": Flash-Next IQ3_XXS (75.8 GB), 78 GB RAM, 2× P100.
  • llama.cpp --moe-cache-mib 12288 -sm layer: 25.5 tok/s. Strata (qr0n's 12 P100 patches, github.com/qr0n/p100-strata, --spec 4): 55 tok/s, prefill 382 vs 157 tok/s at 40K.
  • GSM8K 99 vs 97, HumanEval+ 78 vs 77 (llama.cpp vs Strata). A money calculation was wrong 4/4 on Strata.
  • Whole model must sit in RAM + VRAM; from disk he got 7-9 tok/s.
  • Benchmarks/configs: https://github.com/junkgpulab/strata-vs-llamacpp-p100
• FjX0tXcGzBM "Ternary Bonsai 2 27B in 7 GB: 43 tok/s on One P100": PrismML fork, unpublished kernel patches; one card, 26.6 tok/s stock. Not useful for us until the patches are published.

ISTA-DASLab Flash-Next quants (model card): IQ3_XXS 75.8 GB, task avg 92.57; IQ3_S 83.6 GB, 93.26; BF16 93.12. Shard 2 (28.8 GB n-gram table) is the same in every quant.

#tandem.cpp (checked 2026-11-07)

Source: Code Unpacked video 2TmaeDZ7kEY (2026-11-06; every number is the developer's own) and https://github.com/ewhacc/tandem.cpp (MIT, published 2026-11-01).

• What: one patch (~8.7k lines) on upstream llama.cpp f5222f25c (2026-10-28). It places MoE experts by expert, not by layer: all experts in RAM, leftover VRAM is a cache of hot experts from every layer. Cache misses are split between computing on the CPU and uploading over PCIe (Strata's rule). Plus fused kernels, one CUDA graph per batch shape, MTP head re-quantized 3.6 → 1.8 GiB.
• Claimed: Flash-Next IQ3_S + MTP on one RTX 4090, i5-12400F, 64 GB DDR4: llama.cpp default placement 29.8 → 97.3 tok/s; Strata exact mode 103.6. Same expert cache size as Strata exact: 131 vs 129. Strata's default mode (skips low-ranked experts) still leads: 127 vs 123.6 (LLAMA_MOE_TAIL_SKIP, off by default). Caveat: the baseline is NOT llama.cpp's own --moe-cache-mib, which tandem builds on.
• Quality: LLAMA_MOE_NO_DEFAULTS=1 → KLD 0 vs llama.cpp; defaults → KLD 0.009, 96.1 % same top token, PPL unchanged. Greedy output not repeatable with the cache; a fixed CPU share (LLAMA_MOE_CPU=0.2) makes it repeatable. That is the knob Strata lacks for its money-total problem.
• Runs: -cmoe -ngl 99 --moe-cache-mib auto, -lm none to read into RAM (omit when the model is larger than RAM: 78 GiB IQ3_S runs on 64 GB). Models much larger than RAM lose most of the gain.
• For this rig: untested on Pascal. Tested on sm_89/sm_120 with CUDA 13 only, and CUDA 13 has no Pascal, so we would build with CUDA 12.9 for sm_60 and hope its fused kernels need no DP4A or newer intrinsics. Multi-GPU only with -sm layer (no tensor split), checked for correctness but not tuned. Base is plain upstream, so none of Kmic's P100 kernels. Worth a build test, and a third contender next to Strata and Kmic's --moe-cache-mib once the RAM is in.
• Upstream: PR #30966 (missed experts on the CPU) and draft PR #30971 (adaptive single-graph MoE offload) are open; either could make this land in Kmic's fork later.
• From the transcript (beyond the README):
  • The 64 GB test PC could not hold the 78 GB IQ3_S either; both engines read part of it from the mmapped file on the SSD and still reached 97 tok/s. So 64 GB RAM + our 32 GB VRAM is enough.
  • Also speeds up Qwen3.8's gated-delta-net layers (the same hybrid layers as our 27B).
  • The MTP head re-quantized 3.6 → 1.8 GiB "with the same acceptance"; the freed VRAM goes to the expert cache.
  • Drafting: Strata drafts up to 5 tokens; tandem got slower at 5. With drafting on, Strata leads (148.9 vs 132.4 on DeepSeek-V4.1).
  • 2 or 4 parallel requests work but gain less: it is tuned for one user.
  • Narrator's advice: model much larger than RAM, several users, or non-Linux/NVIDIA → Strata or plain llama.cpp with --moe-cache-mib; otherwise tandem is worth a try.
• Build test on the rig (2026-11-07): builds for sm_60 with CUDA 12.9 after one fix.
  • __nanosleep (sm_70+) in two host-flag wait loops in ggml-cuda.cu breaks the build. Shim: on sm_60 spin on %globaltimer for the same time (/opt/tersler/src/tandem-test/sm60-nanosleep.py, applied by build-test.sh after the patch).
  • The new MoE matvec kernels use ggml_cuda_dp4a; the P100 has no DP4A, so they take the slow byte-wise fallback. Expect less than the developer's gain until that is ported (Kmic uses VMAD).
  • Binary: /opt/tersler/src/tandem-test/llama.cpp/build/bin/llama-server (llama-server only, 11 min at -j4). Not run yet: needs the RAM upgrade for Flash-Next.

#Multi-session test (2026-11-07)

scripts/concurrency-test.py against the pin 790a3eaa1, 27B Q6_K, q4_0 KV, -sm tensor, 150 W. N parallel requests: ~1.5k-token prompt each, thinking off, 768 tokens. tok/s = decode per request / all requests together (wall clock, incl. prefill).

• MTP and parallel slots don't mix in this build. With 2 requests the two together are no faster than one with MTP; with 4 the total collapses to 23 tok/s. The log shows "backend offload failed … using CPU sampler" for every draft slot. Without MTP, batching works: 1.7× total at 4 requests.
• VRAM: 4 slots cost ~2.2 GB per card more than 1 slot at load (per-slot recurrent state, draft contexts), and 4 MTP slots filled the cards completely at runtime. --spec-type none appended after the config's MTP flags does not turn MTP off; the flags have to be left out.
• Prompt reuse (agent replay, 26k-token context with an 18k system prompt):
  • one more turn: 1,171 tokens re-read (5 s); a tool output pruned mid-history: re-read from the last checkpoint (6,596 tokens, 18 s; checkpoints every 8k tokens, --checkpoint-min-step).
  • a subagent with the same system prompt in the same slot: 20 tokens. In another slot: 17,978 tokens, 46 s (cache_n 0): the prefix was not shared across slots, despite --kv-unified and the default 8 GiB --cache-ram. Not investigated yet.
• Takeaway for the agent setup: today it is either one fast session (MTP, ~50 tok/s) or a swarm (4 slots, no MTP, ~17 tok/s each, 51 total). The subagent-prefix miss is the bigger cost: 46 s of prefill per subagent start.
• Why the subagent re-read its prefix (code read, tools/server, 2026-11-08):
  • Partly a test artifact. The test pinned the subagent to slot 1 (id_slot). For a pinned, empty slot, get_available_slot computes f_keep = 0/0 (NaN), so update_cache stays false and the host prompt cache is never consulted. Clients that don't set id_slot (opencode etc.) take the LRU path, which does load from the cache.
  • Real limit 1: cache entries are moved, not copied. server_prompt_cache::load restores the best entry into the slot and erases it from the cache. Of several subagents started at once, only the first inherits the root's prefix; the rest start cold. The root's own entry is gone too, so it later reloads from whichever entry shares its prefix.
  • Real limit 2: only idle slots are shareable. With --kv-unified, idle slots are saved to the host cache and cleared when a new task starts (cache_idle_slots). A slot that is still generating can't seed another one. In-VRAM seq_cp between slots is used only for parent/child tasks (n > 1).
  • Real limit 3: hybrid model. A restored state can only be cut back to a context checkpoint (the delta-net state can't be truncated), so a subagent resumes at the last checkpoint inside the shared prefix.
  • Real limit 4: size. An entry holds the whole slot state: ~0.5 GB at 26k tokens, 6.2 GB at 212k. With --cache-ram 3072, a root context over ~100k tokens can't be cached at all.
  • Fix idea: load a copy and keep the entry (a small change in server_prompt_cache::load), so every subagent can start from the root's prefix. Costs one extra entry of host RAM: after the upgrade.
• Patch install/patches/server-prompt-cache-keep.patch (on the pin, 2026-11-08):
  • Keep a cache entry when a slot restores it (copy, not move; mark it most recently used).
  • With several slots, save a slot to the cache before another request takes it over, even when only its tail is lost (stock: only when more than half would be lost). This was the actual cause: the first subagent took over the root's slot (f_keep 0.92), so the root's state never reached the cache.
  • Before restoring into a slot, save the other idle slots (stock: only after the task launched).
  • A pinned empty slot (id_slot) also looks in the cache (stock: 0/0 skipped it).
  • Swarm test (root 31.5k tokens, then 3 subagents with its 29k system prompt at once, 4 slots, no MTP), stock → patched: subagents 30 / 28,944 / 28,944 tokens read, 196 s → 30 / 30 / 30, 8.5 s. Replay step d (other slot, same prefix): 17,978 tokens, 45 s → 20 tokens, 4.6 s. Root's next turn 13.3 → 16.3 s (copying ~1 GB states to host RAM).
  • Build: /opt/tersler/builds/790a3eaa1-cachecopy (worktree /opt/tersler/src/llama.cpp-cachecopy). Not in production. Needs host RAM for the cache (each 29k-token entry ~1 GB), so after the upgrade.

#Video 0aBLGbfzvUs "6 Local LLMs Ranked on Two Nvidia Tesla P100s" (2026-11-10)

All on 2× P100, llama.cpp (junkgpulab fork), 128K context, q8_0 KV. 126K-token cold read / decode tok/s / own vision / his 21-item rocket-agent checklist (Qwen Code, 3 tries):

Bonus outside the table: Flash-Next on Strata 21/21, ~55 tok/s.

• For us: Ornith 1.5 35B-A3B (MIT, 262K context, ~3B active; official GGUF ornith-ai/Ornith-1.5-35B-A3B-GGUF) is the candidate for the agent/swarm mode: as good as the 27B on his checklist and GPQA, 2.2× faster prefill, faster decode, and with ~3B active it should batch across slots better than the dense 27B. Untested on lean_bench.
• 27B at medium thinking returned empty 3 of 5 times with q4_0 KV; he now runs q8_0 KV everywhere. Second hint (after recipe 2.0) that our q4_0 KV may cause the lean_bench truncations.
• --cache-ram -1 (unlimited) on his 78 GB RAM box: a 126K chat comes back in 2.4 s instead of 6:35. Matches our OOM finding: needs the RAM upgrade.
• Gemma 4's stock template hides tool parameters named description/type (agent loops); fix in the video description. Flint read files outside its working directory 21 times.