Orchestration
Agent Orchestration is at the core of the product and it uses multiple small models (SLMs) to break down a user request to produce comparable results. This includes handling skills, processing complex prompts, and making external tool calls
Model selection is done based on capabilities. The main orchestration model, sl-mini, handles the routing based on tasks.
You can select/deselect a model from the settings page of the desktop app or using the /model command in the terminal user interface (TUI). The orchestrator selects models based on the workload and capability for a given task
How it works
For example, if you have selected gemma4-e2b and qwen2-vl-2b from the list:
When you upload an OCR document or a document with images that contain information, it will select a vision-capable model to process the document. Similarly, if you have a complex skill with multiple steps, it will use a similarly capable model to process and break down the skill into steps and produce the result
Memory
The orchestration process is heavily memory-driven on Apple Silicon-based Macs where GPU memory is shared to take full advantage of all the models selected. For example, if you have enabled gemma4-e2b (2B) and gemma4-e4b (4B) from the settings and your orchestration is a tiny 1B fine-tuned model, the total memory pressure would be 2 + 4 + 1 + 1 (buffer/KV) = 8GB.
We have tested orchestration using NVIDIA RTX 4060 GPU with 8GB memory and it worked comfortably. If your task scope is small and you don't need a larger model, you can turn models on/off based on your needs. Otherwise, the orchestrator will fall back to the default model automatically, which will reduce response quality but works for most regular use cases unless you are processing documents with diagrams, complex math, and images or skills with multiple steps.
Available Models
| Model | Size | Capabilities | Access | Use Case |
|---|---|---|---|---|
| gemma4-e2b | 2B | GENERAL, SKILLS, DP, VISION | Free | Lightweight all-rounder for skills, document processing, and general tasks. Multi-modal, so it also handles images alongside text |
| gemma4-e4b | 4B | GENERAL, DP, VISION | Pro | Larger sibling of E2B for document processing, general-purpose work, and vision-language tasks |
| ministral3-3b | 3B | GENERAL, DP, VISION | Pro | Fast, efficient multi-modal model for document processing and general tasks |
| qwen2-vl-2b | 2B | VISION | Pro | Used exclusively for image processing — a vision-language model specializing in interpreting visual content |
Capabilities map to the work the orchestrator routes: GENERAL for everyday prompts, SKILLS for running skills and making tool calls, DP for document processing, and VISION for images and scanned pages.
The orchestration model itself, sl-mini, is separate from this list — it is always present and does the routing rather than being selected for a task.
Next Steps
We generally use q4_k_m quantization to run models efficiently without much loss on resource-constrained devices.
We will include new models based on use-case and capabilities. If you want us to list a model, reach out to us at hello@smartloop.ai.