Skip to main content
Arka’s LlmFallbackEngine manages all LLM calls across 25 providers with automatic failover, API key rotation, and per-task or per-skill model overrides.

​
Supported providers

Anthropic, OpenAI, Gemini, Groq, xAI, DeepSeek, Moonshot, Z.AI, MiniMax, Venice, Mistral, Cohere, Together, Fireworks, Perplexity, Hugging Face, OpenRouter, Sakana Fugu, Bedrock, Azure, LiteLLM, Ollama, LM Studio, vLLM, and vLLM Cloud.

​
How failover works

The orchestrator builds a candidate list in order:
  1. AI_PREFERRED_PROVIDER + AI_PREFERRED_MODEL (if set)
  2. Live model lists from each provider API (when keys allow)
  3. Built-in chain: Gemini 2.5/2.0 Flash → Groq Llama 3.3/3.1 → Ollama cloud/local
On 429, 401, or timeout, Arka tries the next model. API key rotation (API_KEY_ROTATION=1) cycles backup keys before switching providers.
When you switch providers with arka provider set groq (or ai-pref groq in fish), Arka tries to keep the same model on the new provider before falling back to that provider’s default. To disable this, use arka provider set groq --no-keep-model or AI_KEEP_MODEL_ON_PROVIDER_SWITCH=0.

​
Web chat model order

Interactive web chat, such as the Open WebUI frontend, uses a separate fast chain so replies keep coming when cloud free tiers hit rate limits:
  1. A small local Ollama model, when Ollama is running. Arka picks the first installed model from llama3.2:1b, llama3.2:3b, qwen3:8b, and llama3.1:8b.
  2. Gemini Flash models, when GEMINI_API_KEY or GOOGLE_API_KEY is set.
  3. Groq llama-3.3-70b-versatile, when GROQ_API_KEY is set.
Override the chain or the local model:
Set WEB_CHAT_FAST=0 to use the standard failover chain instead.
Short acknowledgments such as “thanks” or “got it” get a brief welcome. Arka doesn’t continue the previous build.

​
Per-task profiles

Override per task:

​
Per-skill models

Or edit ~/.config/arka/llm-skill-models.json.

​
Benchmark orchestration

If you want model selection to follow live benchmark winners, run a benchmark suite first and then enable orchestration:
The benchmark profile follows the routed task. A request that routes to web_answer uses the chat benchmark results. Agent-style requests use agent.

​
Shared exhaustion cache

Arka keeps one session cache. It skips exhausted models across skills until you reset the cache:
Enable stderr notification on failover: LLM_FALLBACK_NOTIFY=1. Debug attempts: LLM_VERBOSE=1.

​
Local LLM servers

​
Why skills stay concise

Arka does not ask the LLM to memorize 70 skill manuals. Skills are local programs. The model only picks the right name. On LLM route, Arka passes:
  • A compact skill catalog (registered names only)
  • Curated routing rules (high-signal patterns)
  • Shell aliases via ROUTE_ALIASES for your machine
The route task uses low temperature and a dedicated chain so a fast, cheap model handles classification. Heavy lifting runs locally inside the skill.

​
Inspect at runtime

Set SHOW_MODEL=1 (default) to display Model: provider/name under answers.