Goethe
Turn a Linux box with NVIDIA GPUs into a private, OpenAI-compatible endpoint
Emerged finds new products worth a look, daily — what’s new today
If you want a capable 27B model on a single consumer GPU, this is a measured, reproducible recipe instead of weeks of quantisation trial and error.
syv.ai's open serving stack for running Qwen3.8-27B on a single 24 GB RTX 3090 with vLLM. It bundles patches, requantisation scripts and benchmarks: roughly 1,000 tok/s across 64 concurrent requests, around 120 tok/s for a single user with speculative decoding, and 150k–262k context. GSM8K and IFBench numbers sit within a point of the unquantised model. Docker Compose gets it running; a manual path documents every step.
More like this
S
Swift-Qwen3.8-27B
Qwen3.8-27B fine-tuned to think 58% less at near-identical accuracy
MAPLE
Gemma 4 26B cut to 48 experts per layer so it fits a 6 GB laptop GPU
Strata
Runs a 125B mixture-of-experts model on a single gaming PC, fully offline
Nari Qwen3-TTS and Qwen3-ASR
Fast, low-cost speech APIs built on Qwen3 voice models
Warp
Run trillion-parameter models on a laptop by streaming weights from NVMe