ranganborah/specdecode is a project that actually runs speculative decoding — where a small draft model proposes tokens and a large target model verifies them together in a single batch operation to reduce inference cost — on a laptop GPU. The latest commit 8d8626a (2026-10-13) newly adds a 87-line README.md, documenting for the first time how it works, the feature list, the measured figures, and how to get started quickly. The original commit is at https://github.com/RanganBorah/SpecDecode/commit/8d8626ab1ddb2e22303b34e1a5cac0472cd9faa5.
The skeleton of the approach follows the two 2023 papers by Leviathan and Chen, and the README explains that the output probability distribution is the same as when only the target model is used. Inference runs locally through llama.cpp's --model-draft path, and the screen shows a real-time visualization of tokens moving through proposed, verified, accepted, rejected, and corrected, along with the acceptance rate, tokens per second, the number of target calls, and latency. The hand-made draft model CustomModel 5.1 is Llama-3.2-1B-Instruct fine-tuned twice with QLoRA, and if you put .gguf files into the models/ folder you can pick them from the list without any code changes. It is a DevJams'26 entry from VIT Vellore's GDG on Campus 48-hour hackathon, and the stack is React 19, TypeScript, Node/Express with SSE streaming, with PyTorch, PEFT, and TRL on the training side.
However, the figures the README itself reveals point to the fact that this technique does not yet pay off in that environment. On an RTX 3050 Ti laptop (4GB VRAM), with the target set to Llama-3.2-3B-Instruct and the draft set to Llama-3.2-1B-Instruct, both run at Q4_K_M: the target model's forward-pass count dropped from 99 to 27 and the acceptance rate was 75.8%, but the wall-clock speed slowed from 48 tokens per second to 22.7 tokens per second (0.47x). This comes with the explanation that a 3B-class target is already fast and that the overhead of running two models together on a 4GB GPU is more expensive. The README states that this technique gains value with large models on datacenter GPUs where the target is the bottleneck, and the repository records no license or star information. The same conditions can be reproduced directly with npm run benchmarks, and running it requires Node 20 or later, an NVIDIA GPU, and a CUDA build of llama.cpp.