Devotd Github
NewInference & serving

ghguide/self-draft 259203d — Rewrite of the Arm agent demo plan document

On 2026-06-14, ghguide/self-draft uploaded a documentation-only commit that rewrote docs/DEMO_PLAN.md into a within-3-minute timetable and a list of mapped evaluation items for an Arm64 agent memory demo.

Repositoryghguide/self-draft ↗

On 2026-06-14, ghguide/self-draft uploaded a documentation-only commit that rewrote docs/DEMO_PLAN.md into a within-3-minute timetable and a list of mapped evaluation items for an Arm64 agent memory demo.

GitHub description: Zero-download MTP self-speculative decoding for llama.cpp: one flag, ~1.5x lower-latency LLM inference on Arm64 cloud, no separate draft model.

stars
★ 1
Forks
0
Language
Python
License
Apache-2.0
Read original source ↗View on GitHub ↗

In commit 259203d on 2026-06-14, ghguide/self-draft rewrote only the single file docs/DEMO_PLAN.md. It added 26 lines and deleted 48 lines, and touched no code or configuration. The existing document had been a video plan for MTP self-speculative decoding for llama.cpp, and the new document replaced it with an Arm Agent-Memory demo plan whose scenes fit within 3 minutes. The deleted side had contained MTP model file names and an auto-tuning grid, along with self-speculative-specific scenes such as `SPEEDUP: 1.54x`.

The repository description states that it lowers latency by about 1.5x with a single flag in llama.cpp without downloading a separate draft model. The new plan shifted its weight from that feature toward the agent side. It deals with the situation in which a resumed agent recomputes the entire context, that is, the cost of discarding and refilling the KV cache (key/value cache), and it centers on a configuration that restores that cache and immediately emits the first token. It recorded, as on-screen text, 13~91 seconds for recomputation, about 0.2 seconds for the first token after restoration, a 500x reduction in TTFT, and 3.6x the number of agents per GB on a q4 KV basis. It also included instructions to show a bit-exact check verifying whether the sha256 matches, a self-speculative decoding race scene, the handling of the slot manager (slot manager) and `--swa-full`, and a public CI run that builds llama.cpp for Neoverse with KleidiAI i8mm/SVE2.

Submission conditions and a list mapping evaluation items were also appended. Because grading can be done from the video alone, there are instructions to put every on-screen number on display, and submission rules such as within 3 minutes, running on Arm64, in English, public upload, and music without copyright. The 40 technical points were left to the tie-break criteria, the 25 WOW points were told to lead with the 500x figure and bit-exactness, the 20 impact points were mapped to the explanation of 2026 multi-agent workloads and the first llama.cpp/Arm64 port as arXiv:2603.04428, and the 15 DX points were mapped to a one-command copy-and-paste README and free CI reproduction. Filming is to be done on Arm64 hardware (M4 or Graviton), or a public Arm64 CI run may be substituted with a screen capture, and it instructs that model loading be finished in advance outside the camera.

That said, the commit touched only a single document. Figures such as 500x and 3.6x are not measurement results included in this commit but wording to be shown on screen, and their origin is the repository README and the public CI run. The performance basis lies outside this diff, and all that can be confirmed here is the compression of the demo narrative and the documentation of the configuration. Whether the implementation changed must be checked by looking at the code commits separately.