Dense LLMs bigger than your GPU, 3.6× faster

SR-HPM is a llama.cpp patch and a launcher that make speculative decoding pay off when most of a model lives in system RAM. A 27B dense model on a 6 GB laptop GPU goes from 2.7 to ~10 tokens per second, with exactly the same output.

Code on GitHub Paper (PDF) Install

Results

9.6–10.2

tok/s, Qwen3.8-27B on RTX 3060 Laptop 6 GB + Ryzen 7 5800H (stock llama.cpp: 2.7)

extra quality loss: greedy output identical to non-speculative decoding

1.18×

cost of verifying 4 tokens vs 1 on the CPU (llama.cpp: 2.1–2.7×)

~20

ideas that did not work, each with the experiment that refuted it

Decode speed (tok/s) as components are added. Every blue bar produces the model's exact greedy output; the last one trades 0.018 nats/token for +6 %.

How it works

Decoding a model that does not fit in VRAM is bound by reading the CPU-side weights from RAM once per token. Speculative decoding verifies several drafted tokens per pass — but only helps if verifying M tokens costs about as much as one. SR-HPM makes that true on the CPU and then removes every other cost it could measure:

Small-batch CPU kernels. Each weight block is decoded once and applied to all drafted tokens, with a lossless 4-bit re-encoding of IQ3 weights (one pshufb instead of grid lookups) and sign-free paths for offset-coded types.
Cost-aware placement. The GPU gets the layers with the highest CPU cost per byte, plus the output head the drafter shares.
Adaptive drafting. Keep 3 drafted tokens, extend to 7 while the drafter is confident.
Lazy rollback for hybrid recurrent models. Two full states per pass plus exact rank-1 replay, and a split prompt path that keeps them exact at GPU speed (prefill +35 %).
The idle GPU helps verify. It computes a share of the CPU layers' rows, reading them straight from RAM over PCIe.
System details that add up. GPU keep-alive, 2 MB pages, op-offload fix, re-quantization tuned to what the kernels read fastest.

Configuration	Decode	Quality loss vs Q8_0
stock llama.cpp + DFlash2 drafter	2.7 tok/s	KLD 0.043
SR-HPM (exact)	9.6–10.2 tok/s	KLD 0.043
SR-HPM + `--relax 1.0` (opt-in)	~10.4 tok/s	+0.018 nats/token

Install and run

Windows (Visual Studio 2022 C++ tools, CMake, Git, Python; optional CUDA 12 and LLVM):

git clone https://github.com/Javitax47/sr-hpm
cd sr-hpm
powershell -ExecutionPolicy Bypass -File scripts\install_windows.ps1
python -m srhpm.launch model.gguf --draft drafter.gguf --run

Linux:

git clone https://github.com/Javitax47/sr-hpm && cd sr-hpm
bash scripts/install_linux.sh
python3 -m srhpm.launch model.gguf --draft drafter.gguf --run

The launcher reads the model and the machine, picks the plan (all-GPU, MoE or SR-HPM), and starts an OpenAI-compatible server on 127.0.0.1:8080. Works with any dense GGUF model; the large gains need a drafter (DFlash/DFlash2 or a small model of the same family).

What did not work

Half of the project is negative results — each idea was attacked first with the cheapest experiment that could refute it. A few:

Draft trees: ≤ +3 % with the real cost curve; on recurrent models a second chain duplicates the prefix.
2-bit CPU layers: more quality loss (KLD 0.094) and slower — grid decoding is compute-bound.
Activation sparsity with a pass-shared mask: skipping 10 % of weight blocks costs 5–10× the quantization error.
Dropping redundant layers: two layers already more than double the KLD.
Reranking the drafter's 16 candidates without training: the right token is there, but telling which costs as much as verifying it.

The full list, with numbers and scripts, is in the paper and in research/.