Exact, shared-context inference server for small Jev-style decision models
Emerged finds new products worth a look, daily — what’s new today
spinf-serve
Ask a GPU a hundred questions about one document and pay to read it once
Is this yours? Claim itIf you classify content at volume, scoring many questions in a single read with calibrated probabilities can cut both GPU time and the prompt bill.
spinf-serve is a self-hosted inference server for classification rather than chat. It reads a piece of content once and answers every question as a branch of that read, returning probabilities over the options you supply, plus a no-content baseline that exposes the model's bias. It runs Gemma 4 and Qwen checkpoints on a 24 GB NVIDIA card and shares its API with the hosted spinf service. The maker built it to score news for trading and reports about 2.5x vLLM's throughput on a 4090.
More like this
Turn a Linux box with NVIDIA GPUs into a private, OpenAI-compatible endpoint
Open 0.8B to 9B decision models for choice, yes/no and rating questions
Small encoder models that answer routing and triage questions in one pass
Run trillion-parameter models on a laptop by streaming weights from NVMe
Runs a 125B mixture-of-experts model on a single gaming PC, fully offline