The latest release of lateos-ai/reflex is v0.2.9-runpod-hub. This repository is a Rust/CUDA inference engine that handles GGUF models, targeting cold-start latency in situations that require fast responses, such as an agent's real-time decision making. What changed in this version is not stated in the provided material. The README cites a measurement using Qwen3-0.6B-Q4_K_M on a Tesla T4: 483 milliseconds (p50, n=10) from process start to scoring result. This is according to the engine's internal timer, and it explains that measured outside the process, including roughly 110 milliseconds of OS loading, the figure is about 0.6 seconds.
This engine focuses on short paths where a single agent makes a decision, rather than on a server that batches requests to raise throughput. Because CUDA kernels are precompiled with nvcc and bundled into the binary, there is no JIT compilation at run time. It uses a single-tenant structure with no scheduler and no batching, and HTTP functionality is provided by a sidecar separate from the engine. According to the README, in a T4 comparison within the same session, f16 weight storage is 7% faster than f32, uses half the VRAM, and decodes 36–42% faster. It records that on Runpod serverless, engine loading took 1.6 seconds, while the official vLLM worker of the same GPU class took roughly 150 seconds.
However, these measurements are the project's own results from a Tesla T4 and Runpod, and no external verification has been confirmed. The material notes, in particular, that the loading-time comparison with vLLM is dominated by platform provisioning time. So it is difficult to judge a general advantage of the engine itself from these numbers alone. The structure without batching and scheduling also fits narrow uses where cold start matters more than sustained throughput.