Jinfer

An AI inference engine that runs inside the JVM, with no Python sidecar

YSeen on Hacker News

Java teams can ship chat, vision, embeddings, reranking and speech in the same jar as their app, so a Spring service no longer needs a Python process or ONNX runtime beside it.

Jinfer is the inference engine in Quixotic AI's open stack for local AI on the JVM. It loads GGUF and Safetensors models directly, uses pure-Java tokenizers and quantized matrix-multiply kernels, and exposes chat, vision, audio, embeddings, reranking and text-to-speech through integrations for Spring AI and LangChain4j. It compiles under GraalVM Native Image into a single binary, and the tensor layer has backends for CUDA, HIP, Metal and OpenCL. The current release is CPU-only and the author is asking for API feedback. Apache-2.0, 46 stars, 1,886 commits, from Quixotic AI in Switzerland.

More like this

jdBasic

A modern BASIC where you pause a running program, change the code and carry on

Flide

Exact, shared-context inference server for small Jev-style decision models

Edge0

Run large mixture-of-experts models on a Mac by streaming experts from SSD

Yogurt

Meeting notes that stay on your computer, with no bot joining the call

Goethe

Turn a Linux box with NVIDIA GPUs into a private, OpenAI-compatible endpoint

Warp

Run trillion-parameter models on a laptop by streaming weights from NVMe