Native GGUF
Read weights directly from disk with memory mapping and inspect tensors, metadata, and tokenization.
LOCAL-FIRST LLM INFRASTRUCTURE
Local inference.
Made inspectable.
Run GGUF models directly in Rust — with a clear path from mapped weights to generated tokens, a local API, and benchmarks you can reproduce.
GGUFnative loading
Rustone compact stack
HTTPOpenAI-compatible API
rusty-llm — local
$ rusty-llm model.gguf --prompt
"Explain inference in one sentence."
RUSTYLLM
Inference turns your prompt into tokens, runs them through model weights, and predicts the next token — locally.
GGUF ✦ RUST ✦ METAL ✦ LOCAL API ✦ BENCHMARKS ✦ EMBEDDINGS ✦ GGUF ✦ RUST ✦ METAL ✦ LOCAL API ✦ BENCHMARKS ✦
01WHY RUSTYLLM
The shortest path
from weights to insight.
RustyLLM keeps the full inference path visible: GGUF parsing, zero-copy model access, quantized matrix-vector kernels, KV caching, sampling, and HTTP routing.
It is practical enough to run a local assistant — and compact enough to understand, modify, and benchmark without a sprawling framework.
Built for local
Keep models, prompts, and embeddings on your own machine. No hosted runtime required.
Measure the stack
Compare prefill and decode throughput, inspect kernels, and make optimisation decisions from data.
02THE INFERENCE PATH
No mystery layers.
Every stage has a home in the source and a reason to exist.
- 01
Map the model
Open a GGUF file, parse its metadata, and keep quantized tensors close to the original bytes.
GGUF → mmap - 02
Turn text into tokens
Apply tokenizer rules and the model's native chat template before inference begins.
text → ids - 03
Prefill & Decode
Build the KV cache, reuse it across turns when appropriate, then generate one next token at a time.
KV → logits - 04
Put it to work
Use the CLI, local web UI, embeddings API, or OpenAI-compatible HTTP endpoints.
logits → response
03PERFORMANCE, WITH CONTEXT
Fast is useful.
Explainable is better.
RustyLLM exposes separate prefill and decode timings, kernel benchmarks, backend policies, and Metal telemetry — so an optimisation is more than a hopeful toggle.
See benchmark notes ↗Measured local CLI decode, Metal auto, short prompt.
Tune the path your workload actually uses.
04MAKE IT YOURS
Understand local inference from the inside.
Trace a response from GGUF metadata to SIMD kernels and KV-cache reuse — without losing the system in abstraction.
↗Ship a private local assistant.
Start a small HTTP server for chat, completions, embeddings, and familiar OpenAI-style clients.
Benchmark your own machine.
Compare CPU and Metal routes, profile kernels, and keep the data that justified a change.
OPEN SOURCE · MIT LICENSE
Run your first
local token.
Bring a compatible GGUF file. RustyLLM handles the rest — transparently.
cargo install --git https://github.com/SimonWaldherr/RustyLLM rusty-llm
rusty-llm ./model.gguf --prompt "Hello local inference."