Warp
Run trillion-parameter models on a laptop by streaming weights from NVMe
Emerged finds new products worth a look, daily — what’s new today
Phobos
A Rust kernel language that compiles its own GPU code and serves LLMs with it
On an eight-gigabyte 2019 card it matched or beat llama.cpp on five of seven models with nothing installed but the NVIDIA driver, which suggests the vendor math libraries are not load-bearing.
Phobos is a tile-based GPU kernel language written in Rust, plus an inference engine built on it. Every kernel is compiled at runtime from Phobos source: no cuBLAS, no CUDA toolkit, just a driver at R580 or newer. Apache-2.0, 254 commits, v0.1.1 tagged 6 Oct. The author's figures on an RTX 2080 SUPER put decode at 1.05x to 1.82x llama.cpp on five models and behind it on two, prompt processing still slower on small ones; under memory pressure it held three quarters of its decode speed where llama.cpp held thirty per cent. CUDA only. Joa Ebert calls it a learning exercise measured on one card.
More like this
nvkvm-pv
Share one NVIDIA GPU between the host and several KVM guests, no passthrough
PicoLM
A C99 LLM engine that streams weights from disk, from Tru64 to Raspberry Pi
Goethe
Turn a Linux box with NVIDIA GPUs into a private, OpenAI-compatible endpoint
bootgpt
512 bytes of assembly that boot a PC straight into a neural net
Jik
A statically typed language with regions instead of a collector