Vidraft claims its VKAE system can speed up inference of large language models by as much as 23 times on existing GPUs-no server upgrades or additional hardware needed. The company demonstrated this performance on Nvidia’s B200 accelerator. Unlike typical announcements, Vidraft says anyone can replicate these results using a ready-to-run container that includes the model and runtime environment.
The rationale is straightforward and driven by operational costs: training large AI models remains an occasional, costly event. But inference-the process powering cloud AI services, enterprise chatbots, and API platforms-runs nonstop and consumes the bulk of these costs. For years, the industry has focused less on scaling model size and more on maximizing the number of tokens processed per accelerator.
Vidraft ran their tests on Nvidia’s B200, a next-generation AI inference chip unveiled alongside Nvidia’s Blackwell architecture. VKAE delivered multiple times the throughput of baseline query processing systems without noticeable drops in output quality or accuracy. The standout example is Qwen3.5-35B-A3B: under heavy concurrency, the system exceeded 10,000 tokens per second. In more realistic scenarios-with variable context lengths and simultaneous requests-the same model handled around 455 tokens per second. This highlights why inference benchmarks require careful interpretation, as workload profiles can dramatically impact performance.
Details on the exact acceleration technique remain under wraps; Vidraft plans to publish a preprint soon. Meanwhile, they’re offering a container that bundles model weights and an optimized runtime. VKAE supports the OpenAI API, enabling seamless integration into existing deployments without client-side changes. This matters because many teams currently rely on frameworks like vLLM or TensorRT-LLM. Switching to a new system often encounters hurdles not only in raw speed but also in integration costs.
Demand for solutions like VKAE is surging amid a global chip shortage and rising compute expenses. Nvidia’s B200 targets the next wave of AI workloads, yet even cutting-edge GPUs haven’t eliminated the need for software-driven acceleration. This environment has sparked interest in platforms like Groq and Cerebras, which focus on delivering fast, cost-effective AI inference rather than training. If Vidraft’s claims survive independent verification, it could mark a rare case where a single GPU effectively performs like multiple devices.