openllamas/model-xray is a CLI that inspects, along several axes, which model an LLM endpoint is actually running behind. The 2026-11-02 commit adds a feature that attaches a trustworthy same-model reference endpoint and re-measures the quality and tokenizer differences, and it also fixes the defect whereby the overall verdict could come out as a pass even when a sub-item failed.
When a reference endpoint exists, the quality probe pairs each item's primary response with the reference response and compares them. A difference is promoted to a systematic one only when the primary endpoint gets 3 or more items wrong, the relative gap is 25% or more, and the one-sided p-value of the McNemar accuracy test (a statistical test that pairs only the correctness of the same items) is 0.15 or below. If the reference accuracy is 90% or above while the primary endpoint is below 60% and the p-value is 0.05 or below, the quality score is capped at 45; all other systematic gaps are capped at 70. If the gap runs in the opposite direction the p-value rises, which blocks the assertion of the difference, and a difference of a single item is only noted rather than acted on. The quality check runs 12 topic items and 3 control items in the default round, and the number of rounds can be raised to 1–5 with --quality-samples to narrow the 95% confidence interval.
The tokenizer fingerprint probe compares the input token growth vectors of the two endpoints over 7 fixed corpora, revealing differences in tokenizer or model family. A score only comes out if at least 5 corpora are comparable; this item is added with a 5% weight and, if it falls below 40 points, it enters a hard lane that caps the total score at 45. In the same commit, the billing probe gained a short-output path to avoid timeouts caused by long generations, the capability probe distinguishes the model's refusal from parameter constraints, and the context probe uses HTTP 413 responses for size-difference validation. Protocol metadata such as response IDs and systemFingerprint is also collected, but it is not used as the sole grounds for condemnation.
That said, the new feature only works if a reference endpoint is available. The reference model and key are used only for the comparison requests and are configured only in the CLI; the corresponding form is not yet wired up in the WebUI. With all reference settings filled in, calls go from 63 to 89 and input tokens from 240600 to 252600. The README recommends 2–3 repeated runs, because a single run can be skewed by a transient anomaly. It is a change that gains discriminating power only in environments where a same-model reference can be secured.