Complete record of inference optimization research conducted on consumer NVIDIA Blackwell hardware. Systematic autoresearch loops, controlled baselines, logged experiments. All numbers sourced from production hardware and logged with receipts in the public inference-research repo.
All inference experiments run on the cha0tiktower inference node. Single-machine, consumer-tier Blackwell GPU — no NVLink, PCIe interconnect only. Tensor parallelism over PCIe x8+x4 Gen5/Gen4. Full specs at stack.html.
Two RTX 5060 Ti cards serve a 27B in production — Qwen3.6-27B INT4, ~97 tokens a second warm. What gets it there isn't the compute and it isn't VRAM bandwidth — it's two switches most people never flip and one wall NVIDIA built on purpose.
The wall: consumer cards can't do GPU-to-GPU P2P. NVIDIA disables peer access in the driver, so every tensor-parallel all-reduce detours through host RAM instead of going card to card. We measured it — 20.1% of the decode budget, gone. 128 all-reduces a token, 25 microseconds each, pure latency. Not a bandwidth problem. PCIe moves about 250 MB/s during decode, a few percent of even the x4 slot. You don't need a workstation board. You need the lockout gone.
The FP4 silicon is real — the recipe decides whether you touch it. SM_120 has native FP4 tensor cores and vLLM's kernel runs on them. But it only fires for 4-bit activations. The higher-quality W4A16 quants fall back to emulation and eat a 1.7× penalty. The hardware isn't the ceiling. The quantization recipe is.
The two switches: MTP speculative decoding buys +59%. CUDA graphs buy another +60–75%. Turn both off and the box runs at roughly 40% of its number. Most published consumer-Blackwell benchmarks never turned them on — which is why they're low.
The cards aren't the bottleneck. The interconnect and the loader are. Nobody else is measuring this, so here are the receipts.
Inference configurations on cha0tiktower as of 2026-07-08. All clients route through tower:8010 local proxy; Genesis (Qwen3.6-27B INT4) is the current production backend, verified live. Switch command: proxy-switch genesis|nemotron|aeon|ornith|openrouter.
| Configuration | Quantization | Gen | Context | Framework | VRAM | Status |
|---|---|---|---|---|---|---|
| Qwen3.6-27B GPTQ INT4 + MTP n=3 (Genesis) | AutoRound INT4 | ~97 t/s warm (measured 2026-07-08) | 65K | vLLM + Genesis patches | ~28 GB (both GPUs) | active — production (restored 2026-07-03) |
| Ornith-1.0-35B-AEON-Ultimate NVFP4 (Uncensored) | NVFP4A16 (FP4 experts, BF16 attn) | ~101 t/s warm / ~124 t/s peak | 131K | vLLM 0.23 + SM_120 Marlin patch + fp8 KV | ~23 GB (fp8 KV, both GPUs) | trial ended 2026-07-03 |
| Nemotron 3 Nano 30B (MoE hybrid) | Q4_K_M | 117.34 t/s med / 117.60 peak | 32K | llama.cpp cuda128-clean | 24.9 GB (both GPUs) | standby — dated peak 2026-05-20 |
| Qwen3.6-27B PRISM-PRO-DQ (llama.cpp, MTP n=2) | DQ (GGUF) | 39.30 t/s | 32K | llama.cpp | ~28 GB | research eval only — 2026-05-19 |
| AEON (Qwen 3-27B NVFP4) | NVFP4 (NVIDIA ModelOpt) | ~69 t/s | 122K | vLLM 0.19.2rc1 | ~20 GB | stopped |
| Llama 3.3 70B | Q4_0 / 60 GPU layers | 55.6 t/s | 65K | llama.cpp | ~34 GB (OOM ceiling) | historical — VRAM limit |
| Gemma 4 26B (CPU baseline) | Q4_K_M (CPU-only) | 10.4–11.7 t/s | 32K | llama.cpp | — | pre-tower era |
--kv-cache-dtype fp8 is required for the full 131K context window — AEON's original DGX recipe uses BF16 KV (vision tower forces it), which capped context at ~32K on tower. With --language-model-only --kv-cache-dtype fp8, full 131K context loads cleanly at 338K KV token budget. LangChain tool-use eval (84.8%) was measured on the GGUF variant and re-run on the NVFP4 variant 2026-06-29 — overall held at 84.8% (56/66), with a slightly shifted per-task profile (typewriter_1 19/20 vs 20/20, relational 7/8 vs 6/8) (receipt).ggml_cuda_mul_mat_q: invalid argument) on first generation after the prompt eval on SM_120. This is a Blackwell SM_120 + cuda13 MMQ kernel incompatibility — not a model issue.flash_attn.ops module missing from vllm-env — installed flash-attn 2.8.3 with CUDA 13.0 headers; (2) nemotron.service Restart=always kept seizing port 8022 after every stop — fixed with a Restart=no systemd drop-in override; (3) TimeoutStartSec absent from service file — model killed at 90s before completing load. Config upgrades applied: fp8 KV cache (halves VRAM footprint, enables 65K context), qwen3_coder tool-call parser (fixes silent drop bug), explicit SM_120 FlashInfer routing env vars, FlashInfer workspace reduced 413→256 MiB to clear OOM on first MTP inference. Stable at 88 t/s warm. Both Nemotron (117 t/s MoE) and Genesis (88 t/s dense) now available as production backends via proxy:8010.Speed is necessary but not sufficient. Two quality suites validate the production backends across capability dimensions.
The higher-quality quant is measurably higher-quality — and now there's a scorecard. Fifteen domain-expert scenarios, logistics and traffic and restaurants, scored blind by two judges. The NVFP4 W4A16 build beat the INT4 production model 4.13 to 3.73. It's the only model of three tested that didn't scramble the menu-engineering matrix, and the only one that refused to write a severe-allergy client "guaranteed nut-free" from a shared kitchen. A third the speed. That's the trade — latency for judgment. Both judges agreed on direction (in-session Claude +0.40, Haiku 4.5 automated +0.20). Scorecard.
Comprehensive coding/math/instruction/tool/knowledge suite run against vLLM genesis with thinking disabled (enable_thinking=false) for latency. MTP n=3 speculative decoding active. Throughput averaged across 29 varied prompts.
enable_thinking=false for production latency sacrifices deep reasoning chains. The 2 remaining math failures (GSM8K arithmetic, sum-of-multiples) are model errors in no-think mode — not hardware or quantization artifacts. For coding, tool calling, and instruction following the model is production-ready.A curated quality suite validated the then-production Nemotron backend (2026-05-20 campaign; now standby) across five capability dimensions. Each response scored 1–5 by Claude Haiku acting as judge, using per-probe rubrics with expected outputs. Run against the proxy at 117 t/s.
| Probe | Category | Score | Result |
|---|---|---|---|
| r1–r5 (all) | Reasoning | 5/5 each | Syllogism, bat/ball puzzle, sequences, fallacy detection, pill timing — all perfect. |
| f1–f5 (all) | Factuality | 5/5 each | Great Wall myth, 347×28 arithmetic, Canberra capital, 10% brain myth, Berlin Wall year — all correct. |
| i1–i4 | Instruction Following | 5/5 each | Exact list count, raw JSON, two-sentence summary, forbidden-word avoidance — all perfect. |
| i5 | Instruction Following | 5/5 | Exact structured format (DIFFICULTY / TIME / STEPS). Transient empty response on first query; re-queried correctly. |
| c1 | Coding | 3/5 | Vowel-counter function correct; docstring quality below rubric threshold. |
| c2–c5 | Coding | 5/5 each | Bug identification, palindrome with all normalizations, max() one-liner, Fibonacci explanation — all perfect. |
| a1–a3, a5 | Agent Tasks | 5/5 each | JSON extraction, classification array, professional status update, graceful error — all correct. |
| a4 | Agent Tasks | 3/5 | 5-step Python project setup: steps 1–4 correct; step 5 truncated at max_tokens limit (512). |
claude-haiku-4-5-20251001 acting as judge. The two 3/5 scores are both partial-credit results (correct output, minor defect) — zero probes failed outright. The one transient empty response (i5) resolved correctly on immediate retry. Source: randomchaos7800-hub/model-eval.Head-to-head on the four canonical LangChain tool-use tasks from their Benchmarking Agent Tool Use post: two typewriter variants, multiverse math, and relational data. Ornith-1.0-35B (GGUF Q4_K_M) served directly on port 8030 via llama.cpp (build-cuda120-nographs), layer-split across both RTX 5060 Ti 16GB cards, 131k context, flash attention, q4 KV cache. DeepSeek V3.2 via OpenRouter, routed through the tower fleet proxy at :8010 — same hardware, same day, same prompts. Temperature 0. 30-turn hard cap per case. Full agent loop: model calls tool, result fed back, repeat until stop or cap. 66 total cases.
| Task | Ornith (local :8030) | DeepSeek V3.2 (OR :8010) | Avg latency | What it tests |
|---|---|---|---|---|
| Typewriter, 1 tool | 20/20 (100%) | 11/20 (55%) | Ornith 3.6s / DeepSeek 41.5s | Sequential type_letter calls — stop at the right count |
| Typewriter, 26 tools | 20/20 (100%) | 20/20 (100%) | Ornith 2.8s / DeepSeek 12.3s | One parameterless function per letter — pick in order |
| Multiverse Math | 10/18 (56%) | 18/18 (100%) | Ornith 3.6s / DeepSeek 11.1s | Altered arithmetic — must use tools, ignore pretrained math |
| Relational Data | 6/8 (75%) | 8/8 (100%) | Ornith 2.7s / DeepSeek 22.6s | Multi-hop DB-style tool chains over three fake tables |
| Overall | 56/66 (84.8%) | 57/66 (86.4%) | ~3s / ~22s | DeepSeek wins by one point; profiles are inverted |
reasoning_content traces before acting — on single-hop problems this still terminates correctly; on 3+ hop chains (ecoli division, 1+2+3+4+5 via add function, chained logs) it sometimes determines the answer from training and skips tool calls entirely. Two relational failures: one llama-server 500 on a quoted entity name ("Frank The Cat" — malformed tool-call JSON) and one reasoning trace with no tool calls on a multi-hop weather lookup. Mitigation: force tool-use system prompt, raise max_tokens to ≥1024 on multi-hop tasks, sanitize entity names in tool schemas.Throughput benchmarks across task categories, HumanEval code completion, and context scaling from 1K to 32K prompt tokens. Speed suite run on June 25 (GGUF Q4_K_M direct on :8030, dual GPU) and repeated June 27 (production proxy :8010 warm, after NVFP4 upgrade). Context scaling and canonical 512-gen both via prod proxy :8010 on June 27 (NVFP4 variant).
| Task category | GGUF direct :8030 (June 25, gen t/s) | NVFP4 prod :8010 (June 27, gen t/s) | TTFT (June 27) |
|---|---|---|---|
| Short math | 129.7 t/s | 93.9 t/s | 55 ms |
| Medium attention | 127.2 t/s | 92.3 t/s | 117 ms |
| Code / CSV | 126.9 t/s | 123.8 t/s | 48 ms |
| Agentic JSON | 127.5 t/s | 124.1 t/s | 110 ms |
Throughput at increasing prompt lengths. Each point: 128 completion tokens, median of multiple runs.
| Prompt size (tokens) | Gen t/s (median) | TTFT (ms, median) | Elapsed |
|---|---|---|---|
| ~1K (1,227 tok) | 87.2 t/s | 440 ms | 1.75 s |
| ~4K (5,307 tok) | 84.0 t/s | 1,747 ms | 3.45 s |
| ~8K (10,827 tok) | 23.1 t/s | 3,494 ms | 8.09 s |
| ~16K (21,627 tok) | 21.6 t/s | 4,918 ms | 9.12 s |
| ~32K (43,227 tok) | 46.2 t/s | 9,825 ms | 7.48 s |
```python prefix in the returned body) — the GGUF chat template emits fences even on raw completion tasks, causing indentation errors and SyntaxError when the harness strips the first line. Functional reasoning is stronger than the score implies. Confirmed by partial-credit review: logic is correct in most failures; extraction artifacts cause the failures, not semantic errors. Receipts: humaneval-results.json.The pitch: sparse activation should let a big MoE model match a small dense model's speed without dense's quantization tax on quality. Six checkpoints, four quantizations, two runtimes, three unrelated base-model families later — the speed half of that pitch is true. The quality half isn't. Every MoE checkpoint tested beat Genesis on raw throughput. Every one of them lost to it on quality, by a margin the dense-side quantization spread never came close to.
| Model | Runner | Speed | Quality (/5) |
|---|---|---|---|
| GPT-OSS-20B (native MXFP4) | llama.cpp, all-GPU | 133.05 t/s | 2.20 |
| Nemotron-3-Nano-30B-A3B | llama.cpp, all-GPU | 123.4 t/s | 2.80 |
| Qwen3-30B-A3B (Q3_K_M) | llama.cpp, all-GPU | 121.5 t/s | 1.87 |
| Qwen3.6-35B-A3B | llama.cpp, all-GPU | 100.54 t/s | 2.87* |
| — Genesis (dense, INT4+MTP) — | vLLM, production | 97.4 t/s | 3.73 |
| Qwen3-30B-A3B (GPTQ-Int4) | vLLM | 41.66–42.41 t/s | 2.67 |
The starkest result isn't the average — it's the inversion. Qwen3-30B-A3B's fastest configuration (Q3_K_M, everything on-GPU, 121.5 t/s) is also its worst-quality configuration (1.87/5) — the single fastest MoE result of the whole investigation is also the single worst. Nothing on the dense side did that. Speed and quantization aggressiveness traded off gently there; here they inverted.
Two scenarios broke every model the same way. A signalized intersection with a "yellow trap" left-turn conflict, and a roundabout redesign that ignores rail-crossing queue-spillback risk — both failed identically across Qwen, NVIDIA/Nemotron, and OpenAI's GPT-OSS. Three unrelated base-model families, four quantizations, two runtimes, one shared failure shape: a long, confident, well-organized answer that fabricates a plausible wrong mechanism and lands backwards on the one constraint that actually mattered. GPT-OSS-20B — the smallest model tested — failed hardest: it flatly asserted the roundabout's rail-crossing interaction was "None (independent)," the most confident wrong answer of the whole run, and separately told an operator to reheat and re-serve temperature-abused soup. That's not a benchmark score. That's a real failure mode, and it doesn't show up on the dense side at this parameter count.
One real infrastructure bug turned up, and it's a good story. A vLLM 0.25.0 load of Qwen3-30B-A3B-NVFP4 appeared to hang after weight-loading — idle workers, zero GPU utilization, a shm_broadcast warning firing every 60 seconds. First guess: a TP/expert-parallel synchronization stall. Wrong — retested with Genesis's own production NCCL settings, still hung. Second guess: auto-enabled expert-parallel. Also wrong — the flag defaults off and was never set; the log line that looked like evidence of it prints unconditionally regardless. The actual cause, found by checking ps aux instead of treating the hang as a black box: 91 concurrent nvcc/cicc/ptxas processes. FlashInfer has no precompiled kernel for this SM120+fp8_uint4 combination, so it JIT-compiles on first load — uncapped, on an 8-core box, which drove real RAM/swap thrashing, not a deadlock. Capping MAX_JOBS fixed the thrash, but the compile itself still ran past 12 minutes wall-clock. One line in a launch script, and a lesson permanent enough to make case law: an uncapped JIT compile storm looks exactly like a deadlock until you check what's actually running.
No wall turned up that would justify writing Genesis-style custom patches for the MoE path — llama.cpp served every checkpoint cleanly once configured correctly, and it beat vLLM outright for MoE, the opposite of the dense-model result. Verdict: Genesis stays the production pick. MoE is real, sparse activation is real, the speed is real — on this hardware, for this workload, it isn't a free lunch. It's a different bill.
Each campaign runs an automated LLM-driven hypothesis loop — generate config variant, benchmark, evaluate stopping criterion (+5 t/s improvement threshold), iterate. Full logs and raw TSV data available at github.com/randomchaos7800-hub/inference-research.
ggml_cuda_mul_mat_q: invalid argument) after the first inference, every time. The vLLM FP8 TP=2 path also failed — wrong flag (--disable-log-requests removed in newer vLLM) and NCCL engine init failure. Q4_K_M at 117 t/s set the production peak at the time and ran as default backend via proxy :8010 through mid-May; Nemotron is now standby (genesis is the current default).--cpu-offload-gb 1.0 to initialize (Marlin repacking peak creates ~200 MiB initialization overage). But throughput differs by 8×: 5.25 vs 41.33 t/s. Dense 32B: all 32B weights accessed per token, PCIe bus is saturated. MoE 30B-A3B: only ~3B active params per token, PCIe pressure is ~10× lower. NVFP4 was blocked by FlashInfer's CUTLASS SM_120 FP4 kernels requiring CUDA ≥ 12.9; this system ran 12.8 at the time. System upgraded to CUDA 13.0.3 on 2026-06-06 — CUDA version barrier cleared; Qwen3 NVFP4 on SM_120 is now a candidate for re-evaluation.-ngl 99 --flash-attn on -t 1--language-model-only --kv-cache-dtype fp8 restores full 131K context with 338K KV token budget. Kernels: Marlin linear (auto) + flashinfer_b12x MoE dispatch + a local SM_120 Marlin compatibility patch. vLLM 0.23 TP=2 on SM_120 is stable with these patches applied. NVFP4A16 weight format: FP4 quantization on experts/MLP, BF16 on attention layers. LangChain brutal eval re-run pending — GGUF score (84.8%) is the current quality baseline.unknown model architecture: diffusion-gemma). Build target: CUDA 12.8 / sm_120 cuda128-clean. Key flag: --diffusion-kv-cache on is auto-disabled on 2-GPU splits as a conservative default — forcing it on yields +17% and reduces step count from 25 → 22 with no quality degradation observed. The architecture's native strength is longer outputs: per-step cost is nearly constant regardless of canvas size, so n=1024 delivers 2.5× the throughput of n=256. Single-GPU blocked by 300 MB headroom gap (Q4KM needs 16.0 GB, card has 15.7 GB free) — would unlock Flash Attention + KV together for an estimated further 20–30% gain.Generation throughput improvement from baseline across all optimization campaigns. Bars normalized to the largest observed delta (+17% DiffusionGemma 26B kv-cache flag). Negative deltas not shown; failed/OOM campaigns listed separately.
Complete 13-experiment run showing exact variables, deltas, and outcomes. Representative of methodology applied across all campaigns.
| # | Variable | Value | t/s (median) | Delta | Outcome |
|---|---|---|---|---|---|
| 0 | Baseline (Pass 1 canonical, fp16 KV) | — | 74.35 | ±0.00 | BASELINE |
| 1 | VLLM_MARLIN_USE_ATOMIC_ADD |
1 |
80.59 | +6.25 | IMPROVEMENT |
| 2 | VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE |
1073741824 |
73.57 | −0.78 | NO CHANGE |
| 3 | VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE |
805306368 |
74.92 | −0.57 | NO CHANGE |
| 4 | OMP_NUM_THREADS |
2 |
80.14 | −0.46 | NO CHANGE |
| 5 | NCCL_BUFFSIZE |
8388608 |
80.99 | +0.40 | MARGINAL |
| 6 | NCCL_BUFFSIZE |
16777216 |
81.65 | +1.06 | MARGINAL |
| 7 | PYTORCH_MAX_SPLIT_MB |
64 |
81.18 | +0.58 | MARGINAL |
| 8 | PYTORCH_MAX_SPLIT_MB |
128 |
74.44 | −6.15 | NO CHANGE |
| 9 | OMP4 + exclusive buffer (stacked) | combined | 80.54 | −0.06 | NO CHANGE |
| 10 | OMP4 + exclusive + GMU 0.88 (triple stack) | combined | 79.15 | −1.45 | NO CHANGE |
| 11 | ctx 16K + GMU 0.88 | combined | 79.11 | −1.48 | NO CHANGE |
| 12 | Marlin atomic + OMP4 + exclusive | combined | 80.02 | −0.58 | NO CHANGE |
| final | Best config re-verified | — | 80.48 | +6.13 | FINAL |
Operational findings specific to NVIDIA Blackwell SM_120 consumer GPU inference. These characterize the platform envelope for first-generation Blackwell consumer hardware — a dataset that is sparse in the current literature as most published research covers datacenter-class (H100, A100) or older consumer generations (Ampere, Ada).
q4_0 and f16 KV cache only. q8_0, q5_0, and iq4_nl all degrade significantly — they fall back to unoptimized paths with 15–30% throughput loss. This is a first-generation Blackwell constraint; future driver updates may expand coverage. Choosing the wrong KV dtype is one of the most common configuration errors on this platform.
SM_120KV dtype selection
llama.cpp must be built with the cuda128-clean target — not cuda13. The cuda13 build crashes with ggml_cuda_mul_mat_q: invalid argument on the first generation call after prompt eval, on every model tested (Nemotron Q4_K_M, Qwen3 GGUF variants). This is a CUDA MMQ kernel incompatibility between the cuda13 build's quantized matrix-multiply path and SM_120 hardware — not a model issue. The cuda128-clean build routes around this kernel entirely and runs stably. Confirmed 2026-05-20 during Nemotron format trials. Anyone deploying llama.cpp GGUF inference on Blackwell consumer hardware should verify their build target before assuming the model is the problem.
SM_120llama.cppbuild target
VLLM_MARLIN_USE_ATOMIC_ADD=1 enables atomic-add reduce in the gptq_marlin kernel for small-n decode on TP=2 configurations. This produced the largest single-variable improvement in the research: +6.25 t/s (8.4%) on Qwen 3.6-27B without any model or architecture change. The improvement is specific to GPTQ-quantized models running TP=2 over PCIe — it is not applicable to NVFP4 or unquantized deployments.
GPTQ-MarlinTP=2
-ngl 99) on M4, t=1 is strictly optimal — extra CPU threads fight over unified memory bandwidth without contributing to GPU compute. Flash attention is supported on M4 Metal (MTLGPUFamilyApple9) and gives +9–10% on both pp and tg. Metal tensor API is disabled on pre-M5 hardware: llama.cpp logs tensor API disabled for pre-M5 and pre-A19 devices — this blocks the fast tensor execution path and caps generation at ~21 t/s on 7B dense models. MLX is not a drop-in replacement: despite being Apple-native, it is 3× slower on prefill than llama.cpp Metal for Q4_K_M weights — marginal generation advantage does not compensate. Fanless thermal throttle is a hard constraint: sustained benchmarking degrades pp from 209 t/s (cold) to 152 t/s (warm, 20 min) — a 24% loss. First inference session each day is significantly faster than sustained use. KV V-cache quantization is blocked on Metal flash-attn: any V-quantized configuration fails completely; K-only quantization is marginal (+0 to +3%). Established 2026-05-22 on MacBook Air M4, llama.cpp 9270 (Homebrew), macOS 15.
MetalApple Siliconfanless thermalpre-M5
ModuleNotFoundError: No module named 'flash_attn.ops'. Build requires CUDA_HOME=/usr/local/cuda-13.0 and --no-build-isolation; flash-attn 2.8.3 supports torch 2.11.0+cu130. (2) Port isolation is mandatory: any co-resident inference service (llama.cpp, another vLLM) must have Restart=no in its systemd unit. A service with Restart=always will re-claim the inference port within seconds of a genesis stop — causing a phantom "address in use" crash loop that looks like a genesis bug. (3) TimeoutStartSec=300 is required in the systemd unit file. The default (90s) is shorter than the model load time — systemd kills the process mid-load with no error other than "start operation timed out." Absent this, genesis appears to hang when it is actually loading normally.
SM_120genesisflash_attnsystemd
--diffusion-kv-cache is auto-disabled on 2-GPU splits in PR #24423 as a conservative default. Forcing --diffusion-kv-cache on is safe on Blackwell TP=2 and delivers +17% throughput with fewer steps (25 → 22) — the largest single-flag gain found in this research. Single-GPU path: Q4_K_M requires 16.0 GB; RTX 5060 Ti has 15.7 GB free after driver overhead. 300 MB gap prevents single-GPU load. Single-GPU would enable Flash Attention + KV cache simultaneously — estimated 20–30% further gain. Requires Q3_K_M, which needs the BF16 GGUF (~50 GB) as quantization source.
block diffusioncanvas scalingDiffusionGemma
Gemma4UnifiedForConditionalGeneration support — they fall back to a generic Transformers wrapper that (a) carries 230 MiB of overhead that blocks fp8 single-GPU loading by exactly 114 MiB, and (b) does not support tensor parallelism, making TP=2 fail on weight reshape. The fp8 miss is a near-miss: the model loads to 14.89 GiB on a 15.47 GiB GPU before the profile run OOMs. Native engine support lands within weeks of a model release; the evaluation cadence should be: test llama.cpp first (always works), then revisit vLLM/SGLang after the next PyPI release. The HF model artifacts are already on disk and require only a service restart when engine support ships.
new modelengine readinessTransformers fallback
Historical baseline from cha0tikhome (Beelink EQI12, i5-1235U, 32 GB DDR4). Establishes the starting point before the RTX 5060 Ti tower arrival April 17, 2026. CPU inference ran production workloads for the first phase of the research.
| Metric | CPU Era Value | GPU Era Value | Ratio |
|---|---|---|---|
| Gen throughput | 10.44 t/s (avg) | 83–117 t/s | 8–11× |
| Throughput range | 5.02–11.72 t/s | 71–117 t/s | 7–23× |
| Time to first token | 616–2,224 ms | <200 ms | 3–11× |
| Swap usage | 4.6–7.7 GB chronic | 0 GB | eliminated |
| Model size supported | ≤26B (VRAM-limited) | ≤70B (at Q4) | 2.7× parameter scale |
| Context window | 8–16K (swap risk) | 128K (fp8 KV) | 8–16× context |
Reference benchmark for consumer Apple Silicon. MacBook Air M4, 16 GB unified memory, llama.cpp b200 (Homebrew) via Metal backend. Establishes the ARM/Metal data point for comparing inference efficiency across hardware tiers. Model: Qwen2.5-7B-Instruct Q4_K_M (4.36 GiB, llama-bench standard test). Run: 2026-05-22.
| Test | M4 MacBook Air | RTX 5060 Ti (tower) | Ratio |
|---|---|---|---|
| pp512 | 207.39 ± 3.61 t/s | ~900–1,100 t/s | 4–5× |
| tg128 | 21.37 ± 0.13 t/s | 73 t/s (dense Q4) | 3.4× |
| Model | Qwen2.5-7B Q4_K_M | Nemotron 30B Q4_K_M | — |
| Memory | 16 GB unified | 16 GB GDDR7 (single GPU) | — |
| Backend | Metal (MTL,BLAS) | CUDA 13.0.3 | — |
| Tensor ops | disabled (pre-M5) | full CUDA graph | — |
| Power envelope | ~15 W TDP | ~165 W TDP | 11× difference |
tensor API disabled for pre-M5 and pre-A19 devices on M4. The Metal tensor execution path (introduced for M5/A19) is not available — Metal Shader Language kernels run the standard path. This is the primary reason M4 generation throughput tops out at ~21 t/s for a 7B dense model, vs the M5 which unlocks the tensor path. Upgrading to an M5 MacBook would push this to ~30+ t/s on the same model. The prefill rate (207 t/s) reflects Metal compute shader efficiency, which is not blocked by this constraint.Second Apple Silicon data point, different machine and framework than the MacBook Air baseline above: Mac mini M4, 16 GB unified memory, MLX (not llama.cpp) — Apple's own framework, which generally beats llama.cpp's Metal backend on M-series unified memory. Quant hunt across four candidates on the same 15-scenario domain-expert suite (logistics/traffic/restaurants) used elsewhere on this page, judged directly against the suite's rubrics.
| Model | Recipe | Quality (/5) | Speed | Verdict |
|---|---|---|---|---|
| genesis (tower, reference) | Qwen3.6-27B GPTQ INT4 | 3.73 | ~97 t/s (dedicated GPU) | same base as Bonsai, light quant — clean reference point |
| Ornith-1.0-9B-4bit | MLX uniform 4-bit, 9B | 3.00 | 18.66 t/s | live default |
| Ternary-Bonsai-27B | Prism ML ternary, 1.71 bpw | 2.93 | 12.16 t/s | not adopted |
| Bonsai-27B | Prism ML binary, 1.125 bpw | 2.47 | 20.33 t/s | not adopted despite raw speed |
All benchmark data is public. Experiment logs, TSV results files, autoresearch scripts, and optimal configuration shell scripts are in the inference-research repository.
| Artifact | Location | Contents |
|---|---|---|
| Mac mini quant hunt receipt | model-eval/results/domain-suite-mini-quant-hunt-20260716.md | Full scorecard + failure-mode analysis, Ornith-9B vs Bonsai-27B ternary/binary vs genesis reference |
| inference-research repo | randomchaos7800-hub/inference-research | All autoresearch scripts, TSV results, optimal config scripts, research logs, incident reports |
| autoresearch scripts | autoresearch-*.py |
9 LLM-driven optimization loops — vLLM (27B, AWQ, AEON, GLM4), llama.cpp (70B, SuperGemma, MoE, Beelink) |
| M4 autoresearch log | autoresearch-qwen25-7b-m4-log.md |
Full 25-experiment sweep log: thread sweep, flash-attn, MLX vs llama.cpp, Qwen3-8B comparison, KV cache sweep, thermal findings |
| M4 results TSV | autoresearch-qwen25-7b-m4-results.tsv |
25-row experiment table: config, pp512 t/s, tg128 t/s, notes — Apple M4 MacBook Air 2026-05-22 |
| TSV results | autoresearch-*-results.tsv |
Per-experiment: variable, value, tg_median, tg_p90, delta, outcome, description |
| optimal config scripts | current-best-flags-*.sh |
Production launch commands for all models with all optimal flags applied — includes M4 Metal config |
| model queue | model-queue.md |
Current deployment state, known constraints, tested model inventory |
| vLLM incident report | vllm-upgrade-incident-2026-05-10.md |
Full post-mortem: 0.19.2rc1 → 0.20.2 failure on SM_120, NCCL crash, kernel pin, recovery procedure |
| Research paper | papers.html | Preprint: Commodity Hardware and Persistent AI Companions — Zenodo CC BY 4.0 |
| Full stack specs | stack.html | Hardware configuration, agent infrastructure, service topology |