Club-3090 Brings Experimental FP8 Quantization to Qwen3.6-27B, Enabling Near‑BF16 Quality on Dual RTX 3090 Rigs

The latest experiment from the club‑3090 repository marks a tangible step toward making frontier‑scale language models runnable on modest, locally‑owned hardware. By adding FP8 support for the Qwen3.6‑27B model, the project shows that a quantized version can retain the performance of the full‑precision BF16 baseline while fitting comfortably into the memory of dual RTX 3090 cards [1]. This development is noteworthy not only for its technical merit but also because it reinforces the “off the thumb” ethos: powerful AI no longer requires a subscription to a hyperscale cloud; it can be self‑hosted, tweaked, and deployed on equipment many enthusiasts already possess.

FP8, an 8‑bit floating‑point format, strikes a balance between the expressive range of BF16 and the compactness of lower‑bit integer schemes. The club‑3090 team reports that the official Qwen/Qwen3.6‑27B‑FP8 model “performs virtually identically to the original unquantized BF16” [1]. In practice, this means that the model’s perplexity, generation quality, and ability to follow complex prompts remain essentially unchanged, yet the memory footprint drops enough to allow the model to be loaded simultaneously on two GPUs without resorting to aggressive off‑loading or model parallelism tricks that often introduce latency or complexity.

For practitioners who have been juggling multiple models across heterogeneous rigs, the implication is immediate. A typical workflow described in the community—using a MacBook Pro M1 Pro with 32 GB to run a Qwen3.5‑35B coding agent while reserving a gaming PC equipped with an RTX 5070 Ti for heavier tasks [2]—can now be reconsidered. With FP8‑quantized Qwen3.6‑27B, the same dual‑RTX 3090 setup can host a single, large model for both coding assistance and general conversation, freeing the secondary machine for other workloads or for running a second, smaller specialist model. This aligns with the growing interest in “two LLM combo” strategies, where different models are assigned to complementary roles [2].

The memory savings also speak directly to a persistent pain point in multi‑GPU serving setups. Users of llama‑server in router mode have observed that a model pinned to a single GPU still grabs a CUDA context on every card, leading to out‑of‑memory errors when other accelerators are already saturated [3]. By reducing the per‑GPU memory demand, FP8 quantization alleviates pressure on the CUDA context allocation, making it easier to keep multiple models resident without triggering OOM conditions. While the router issue itself remains a scheduling challenge, the lower memory profile of FP8‑quantized models reduces the likelihood that a single model will exhaust resources across the fabric.

Beyond the immediate FP8 experiment, the broader quantization landscape is evolving in parallel. Recent work has merged NVFP4 support into llama.cpp, opening another low‑bit pathway for models such as Gemma 4 QAT [15]. Simultaneously, discussions around 2‑bit quantization‑aware training (QAT) hint at future extremes where even larger mixture‑of‑experts (MoE) models could fit into consumer‑grade RAM [8]. These efforts collectively point to a trend: the community is actively probing how far precision can be trimmed before quality degrades, and FP8 sits at a sweet spot where the trade‑off remains negligible for many use cases.

Benchmarking provides a sanity check for any quantization effort. The Qwen 3.6‑27B model’s performance on the DeepSWE benchmark—scoring roughly 2 % (1.79 % rounded up) and placing it above Haiku 4.5 and Minimax M2.7 [5]—offers a baseline for reasoning‑oriented tasks. Because the FP8 variant is reported to be virtually indistinguishable from BF16, we can expect similar scores when the quantized version is evaluated on the same suite. This consistency is crucial for adopters who need reliable, predictable behavior when swapping precisions in production pipelines.

Hardware trends further amplify the impact of such software advances. The recent unveiling of the GMKtec EVO‑X3 motherboard, which packs OCuLink, Wi‑Fi 7, and dual PCIe 4.0 slots while teasing a forthcoming 192 GB Ryzen AI MAX+ 4

Sources

1. club-3090 adds experimental FP8 support for Qwen3.6-27B!
2. Preferred two LLM combo
3. llama-server router: a model pinned to one GPU still grabs a CUDA context on every card, so it OOMs when my others are full. Am I missing a flag or is this just how it is?
4. Qwen 3.6 27B on DeepSWE
5. 2-bit QAT model releases
6. NVFP4 on llama.cpp?