Local inference Pure Go GGUF Embeddings

Your models.
Your machine.
Your data.

GopherLLM is a compact GGUF runtime written in Go. Run language and embedding models locally, expose familiar APIs, or embed inference directly into your own Go application.

Start locally Explore the API

MIT licensed. No Python runtime. Local by default; remote features remain opt-in.

Quickstart

# Build the CLI
git clone https://github.com/SimonWaldherr/GopherLLM
cd GopherLLM
make build

# Discover local GGUF models
bin/gopherllm --model-dir /path/to/models --list-models

# Start the API and private chat UI
bin/gopherllm --model-dir /path/to/models \
  --model "model-name-or-file-fragment" \
  --serve 127.0.0.1:8080 --chat

Chat at 127.0.0.1:8080/chat

Localinference stays on your machine

OpenAI + Ollamacompatible local APIs

Pure Gono third-party Go modules

GGUFmmap-backed, zero-copy weights

Built for local work

Everything between a GGUF file and a useful application.

One runtime covers model loading, generation, embeddings, model inspection, chat, tools, and production-compatible HTTP endpoints.

Private browser workspace

Use the embedded chat interface for persistent local conversations, smart context windows, model switching, branching, file attachments, and export.

See the server UI

Fast model loading

Memory-mapped weights, zero-copy quantized tensors, split GGUF discovery, and parallel page warm-up keep startup predictable.

Drop-in APIs

OpenAI-compatible and Ollama-compatible routes make existing clients easy to connect, while native endpoints expose GopherLLM features directly.

Hardware-aware runtime

AVX2, FMA, ARM64 NEON, and optional Metal kernels accelerate the same portable Go inference graph.

Tools, skills, and retrieval

Build agentic workflows with function calling, local skills, Wikipedia and Wikidata research, embeddings, and browser-side RAG over saved conversations.

Explore agent features

Retrieval stays local

Embed documents where they live.

Run BERT, Nomic-BERT, and Granite Embedding GGUFs in the same small runtime as your chat model. No Python service or remote embedding call is required.

Vectors are mean-pooled and L2-normalized, so dot product is cosine similarity. Use the CLI, the OpenAI-compatible endpoint, or the Go library—whichever belongs in your stack.

Explore embeddings API See supported models

Local embedding flow

# Create a normalized vector locally
bin/gopherllm /path/to/embedding-model.gguf \
  --embed --prompt "semantic search query"

01
Load a compatible GGUF with memory-mapped weights.
02
Embed queries and documents with the same local model.
03
Store or compare the resulting vectors in the system you already use.

Broad model coverage

Use the model that fits the job.

GopherLLM validates each architecture and tensor layout instead of treating unknown files as generic Llama models.

General and instruction

Llama 2 and 3MistralMinistralMixtral Gemma 1–4Phi 2–4GraniteStableLM

Reasoning and coding

Qwen 2–3Qwen 3.5–3.8 (experimental)Qwen Coder DeepSeek V2 and V3 Kimi K2GPT-OSSEXAONE 4InternLM2

MoE, recurrent, and embeddings

Qwen MoEGemma 4 MoENemotron-H Mamba2BERTNomic-BERTGranite Embedding

Review the full compatibility matrix

Qwen 3.5–3.8 hybrid GGUFs are currently experimental, text-only paths.

Connect your stack

Familiar on the outside. Native Go underneath.

Keep the client you already use, generate local vectors, or call the runtime directly.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [
      {"role": "user", "content": "Explain mmap in one sentence."}
    ],
    "stream": true
  }'

curl http://127.0.0.1:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [
      {"role": "user", "content": "Explain mmap in one sentence."}
    ]
  }'

curl http://127.0.0.1:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "input": [
      "semantic search query",
      "document to compare"
    ]
  }'

ctx := context.Background()
model, err := gopherllm.Open(ctx, "model.gguf")
if err != nil {
    log.Fatal(err)
}
defer model.Close()

result, err := model.Generate(ctx,
    "Explain mmap in one sentence.",
    gopherllm.WithMaxTokens(80),
)
if err != nil {
    log.Fatal(err)
}
fmt.Println(result.Text)

Performance without guesswork

Measure the machine you actually run.

For generation models, Auto Mode measures thread counts, activation paths, KV-cache formats, and prefill chunks on the machine you actually use. Results are cached and applied on the next run.

Read about auto mode

Memory mapped

Weights stay backed by the GGUF file and quantized tensors borrow mapped memory directly.

Batch-friendly embeddings

Dedicated BERT, Nomic-BERT, and Granite embedding paths keep long local documents moving efficiently.

Compact KV cache

Optional f16 key-value storage reduces memory pressure while optimized kernels preserve throughput.

Portable acceleration

Architecture-specific assembly and Metal paths sit behind the same Go API and test suite.

Run local by default

Start with one GGUF file.

Build the binary, point it at your model directory, and choose the interface that fits your application.

Open the quickstart Read the Go docs