Qwen 3.6 27B Speculative Decoding Hits ~100 TPS on a Single RTX 3090

A fresh benchmark posted to r/LocalLLaMA shows that the Qwen 3.6 27B model, when run with speculative decoding, can sustain roughly 100 tokens per second on a lone RTX 3090 [1]. The test was carried out on a Xeon E5‑2666v3 CPU paired with 64 GB of RAM and a single 24 GB RTX 3090 GPU, comparing five inference engines—three llama.cpp forks, the mainline llama.cpp, and Lucebox—across two quantization schemes. The benchmark script from the community‑maintained “club‑3090” repo was used, with long prompts built from en8wiki via two simple helper scripts [1].

Speculative decoding works by pairing a small, fast “draft” model with the target large model. The draft proposes several candidate tokens in parallel; the target model then verifies them in a single forward pass, accepting correct guesses and falling back only when the draft errs. This reduces the number of costly large‑model passes needed per output token, effectively boosting throughput without sacrificing quality. The reported ~100 TPS places Qwen 3.6 27B in a range previously seen only with much larger GPUs or multi‑card setups, highlighting how algorithmic advances can bring high‑speed LLM inference to consumer‑grade hardware.

This result resonates strongly with the “off the thumb” perspective that values independence from big‑cloud providers and the ability to run powerful models locally. When developers can achieve near‑real‑time generation on a single RTX 3090, the barrier to self‑hosting drops dramatically—especially for use‑cases like coding assistants, chatbots, or edge‑device agents where latency and cost matter. The achievement also underscores that raw model size isn’t the sole determinant of usability; clever inference techniques can unlock performance that would otherwise require expensive data‑center GPUs.

Supporting developments in the ecosystem reinforce this momentum. Hugging Face’s new hardware‑compatibility filter lets users quickly surface models known to run well on specific GPUs, making it easier to pair a model like Qwen 3.6 27B with the right runtime [3]. The availability of GGUF builds—such as Bartowski’s DS4‑GGUF release [5] and the MTP‑only subsets for Qwen 3.5/3.6 [6]—further streamlines local deployment by providing formats optimized for llama.cpp and related engines. Meanwhile, NVIDIA’s own NVFP4‑quantized Qwen 3.6‑27B [7] shows that hardware vendors are also pushing low‑bit representations that dovetail with speculative decoding to squeeze more tokens out of existing silicon.

For contrast, consider the recent Huawei OpenPangu‑2.0‑Flash release [4], a 92 B‑parameter Mixture‑of‑Experts model with only 6 B active parameters. While impressive in scale, its deployment still demands substantial memory and compute resources, positioning it more as a research‑grade artifact than a drop‑in solution for a single consumer GPU. The Qwen 3.6 27B benchmark, by contrast, demonstrates that a mid‑sized model can deliverable hardware slice that many developers already own.

Even older GPUs remain viable: a recent Tesla V100 16 GB NVLink benchmark showed that bandwidth‑heavy Volta cards can still handle respectable LLM workloads [12]. Yet the RTX 3090’s combination of ample VRAM, high memory bandwidth, and mature software support makes it a sweet spot for pushing speculative decoding to its current limits.

From a research standpoint, the result invites further exploration of draft‑model design, better quantization synergies, and adaptive speculation depth. It also serves as a reminder that, while frontier models continue to grow in size, the practical impact of AI often hinges on how efficiently we can run them where they’re needed—whether that’s a laptop, a workstation, or an edge node.

In short, the Qwen 3.6 27B speculative decoding benchmark isn’t just a performance number; it’s a tangible step toward democratizing high‑throughput LLM inference. By coupling algorithmic ingenuity with accessible hardware, it expands the realm of what’s possible without relying on massive cloud allocations

Sources

1. Qwen 3.6 27B Speculative Decoding Bench: Pushing ~100 TPS on a single RTX 3090
2. NEW on Hugging Face: Filter by hardware compatibility
3. Huawei open-sources OpenPangu-2.0-Flash - 92B total,6B active
4. Bartowski has delivered DS4 GGUF
5. MTP-only GGUF subsets: Qwen3.5/3.6
6. nvidia/Qwen3.6-27B-NVFP4 just dropped
7. Tesla V100 16GB local LLMs, single and dual NVLink benchmarks