tiny-agent

Engines & offline inference

tiny-agent supports two inference backends: 100% private local models via Ollama, and high-throughput cloud inference via Groq.

Ollama (Local Offline Setup)

In local mode, your source code, prompts, and tool outputs never leave your machine. Pull your preferred coding model and start tiny-agent:

$ollama pull qwen2.5-coder:latest
$tiny-agent --local

Recommended models for local coding:

  • qwen2.5-coder:latest (Default, excellent tool-calling)
  • qwen2.5-coder:14b (Deeper reasoning for larger refactors)
  • llama3.2:3b (Ultralight, fast on laptop hardware)

Groq (Cloud Inference)

Groq provides ultra-fast LPU inference (500+ tokens per second), enabling instant tool call streaming and immediate turnaround:

bash
# Set your Groq API key
export GROQ_API_KEY="gsk_..."

# Run cloud inference using openai/gpt-oss-120b
tiny-agent --cloud

Default Groq model is configured to openai/gpt-oss-120b with a low temperature of 0.2 for deterministic tool invocation.

Single binary compilation

Compile tiny-agent into a standalone executable using Bun's native binary compiler:

$bun run build

Produces a self-contained executable under dist/tiny-agent with sub-10ms cold start time and zero dependencies.