Phobos
A Rust kernel language that compiles its own GPU code and serves LLMs with it
Emerged finds new products worth a look, daily — what’s new today
Warp lets you run a 552B model in under 5 GB of RAM at nearly 4 tokens a second by paging weights from disk, so frontier-scale models become testable on hardware you already own.
Warp (Weight-Aware Runtime and Paging) is a C inference engine from SQLite AI that streams model weights from NVMe storage instead of loading them into memory. On a 64 GB MacBook Pro it runs the full 2.78-trillion-parameter Kimi K3 at about 0.6 tokens a second, and DeepSeek-V4.1-Flash (552B) at 3.8 tokens a second in 4.9 GB of RAM. No third-party dependencies, embeddable as a library, text plus images for supported models. Apache-2.0, 2.4k stars, 180 forks, 302 commits. An HN commenter fairly notes that under 4 tokens a second is slow next to a cloud API.
More like this
Goethe
Turn a Linux box with NVIDIA GPUs into a private, OpenAI-compatible endpoint
Jinfer
An AI inference engine that runs inside the JVM, with no Python sidecar
Edge0
Run large mixture-of-experts models on a Mac by streaming experts from SSD
InstinctFlash
A serving runtime that speeds robot policy models up for real control loops
ai-infra-book
An open textbook that sizes LLM inference and training from first principles