Orkas Foundation Refactor

When an AI agent product matures, the most expensive thing isn't features — it's the foundation. This article walks through the ground-up refactor Orkas did across its 1.0 release line — a full overhaul of model invocation, the agent loop, multi-agent orchestration, and the tool ecosystem — and the trade-offs behind each decision.

The short version What the rewrite was for The result is the app you can install today: open source under MIT, local-first, and running on your own provider keys if you want.

Download Orkas — free

Why touch the foundation

Orkas is a local-first desktop AI agent workspace: all agent work runs inside a process on the user's own machine, data lives locally, and end-to-end cloud sync happens on demand. Features piled up fast in the early versions — a skills library, a knowledge base, connectors, group-chat-style multi-agent — but the further we went, the clearer it got: the real bottleneck wasn't any single feature, but three "foundation-level" things.

If the model-invocation layer follows the conversational old road, it gets shackled by a string of wrong assumptions. Calling a large model as if it were "one-question-one-answer chat" smuggles in a pile of defaults that make sense for chat but not for an agent: a fixed output-token cap, serial tool calls, hidden timeouts, a single hard-wired provider. An agent is a long-running flow that runs dozens of turns in a row, routinely brushes up against the context window, needs to read files in parallel, and can be interrupted by the user at any moment — every one of those defaults will bite you in production. Worse still, the most valuable capabilities of a desktop agent — fine-grained file operations, local search, running a shell, parallel multi-worker execution, long-horizon task solving — are exactly the ones that this layer of assumptions shuts out.
Orchestration was "static planning." The early version was a plan/DAG engine: first have the model break the task into a plan graph, then have an executor dispatch by the graph. It sounds tidy, but the reality of an agent is highly dynamic — reading one file reveals you need to change direction, and the result of one sub-task decides who the next step goes to. Freezing decisions into a pre-generated graph means every "the plan can't keep up with reality" has to be patched inside the executor.
The ecosystem was a closed catalog. Skills could only come from the official marketplace, connectors were a hard-coded catalog, and the external agent tools already on the user's machine were a complete black box to Orkas. A user wanting to plug in a third-party project, their own MCP server, or have an agent already on their machine call back into Orkas's skills and knowledge base — architecturally, none of it was possible.

The thesis of this refactor is simple: take the agent's foundation back into our own hands. Concretely, it lands as four interwoven through-lines — a self-built in-process runtime, a provider-agnostic model layer, dynamic group-chat orchestration, and the move from a closed catalog to an open host. Let's go through them one by one.

1. Bringing a full coding-agent capability set onto the desktop

The desktop is the agent's home turf — here there's a real file system, a real shell, a real local toolchain. An assistant that can only chat is wasting the environment; what actually leverages the desktop's advantage is a complete coding-agent capability set: file reads and writes down to the character range, cross-file search, running bash and system tools, spinning up multiple workers in parallel, and the long-horizon problem-solving to keep running dozens of turns until a complex task is genuinely finished.

The core deliverable of the refactor exists precisely to bring this capability set natively into Orkas's own process — a standalone, dynamically loadable, in-process agent runtime (called core-agent in the code). It isn't yet another chat wrapper; it's an agent engine Orkas controls itself.

The key architectural decision was to split it into two layers:

Engine layer (standalone package): pure agent machinery — the tool-calling loop, streaming events, context compaction, error classification and retry, the provider abstraction, the sandbox, skill scanning, memory, self-evolution. It knows nothing about any Orkas business: it doesn't read business data directories, doesn't understand the conversation file format, and never touches IPC.
Adapter layer (inside the main process): wires the engine into Orkas — session persistence, provider rotation, tool permissions, the skill registry, connectors, the knowledge base, the various generation tools. It translates the engine's native events into Orkas's own event shapes, so the business layer only ever sees a stable interface.

This engine/adapter boundary is the root of all the flexibility that follows. The engine can be tested and evolved independently; the adapter layer can safely absorb Orkas-specific complexity (rotation, cooldown, sandbox, permissions) without polluting the engine. The team's architecture review summed it up in one line: this is earned complexity — don't go merging it.

What the capability set actually is

Holding the runtime in our own hands isn't about showing off; it's about letting the agent genuinely "get its hands dirty" on the desktop. The capability set falls into roughly four groups:

Fine-grained file operations and local search. read_file supports reading by character range and auto-extracts text from PDF / Office documents; edit_file does precise "old string → new string" replacement and requires a read before any write; write_file lands the artifact and keeps an account of it; stat_file probes size; search_files locates by name/glob, grep_files searches content across files. This group lets the agent "dig through code, change files" in a real workspace like an engineer, instead of only being able to swallow and emit whole blocks.
Bash and system tools. A sandboxed shell executor, with a background-execution mode (long tasks detach from the current turn, logs go to a file) and risk-graded gating for dangerous operations. A large part of a desktop agent's leverage comes precisely from being able to directly command the system toolchain.
Parallel multi-worker. Within a single turn, independent read-only tools run concurrently; at the task level, the commander can also fan out independent sub-tasks to multiple workers in parallel (see Section 3). Parallelizing where it's safe to is the key to compressing "long-horizon tasks" down to acceptable wall-clock time.
Long-horizon reasoning and task solving. A loop that can run dozens of turns in a row, manage its own context, recover from errors, and never get stuck spinning in place — this is the dividing line between "finishing a complex job" and "answering a question."

How it was made production-grade in engineering terms

"Writing your own loop" sounds like asking for trouble, and it does carry maintenance cost. But what it buys is fine-grained control over the agent's entire lifecycle. That control isn't abstract — it's a set of concrete improvements, each one corresponding to whether one of the capabilities above "holds up" in production:

Real context window + compact only at 80%. The engine reads each model's real context window (including million-token-window models) and triggers compaction only when usage hits 80% — rather than conservatively starting at 60% and throwing away 40% of the useful context. There's also a "no-gain compaction" guardrail: if the retained tail already fills the window (say, a very large file-read result sitting in the tail), compaction can't free anything, so it just logs a warning and skips, never spinning a wasted summary call. Whether a long-horizon task can "remember what came before" all comes down to this.
Adjacent read-only tools in parallel. When the model fires off read-file, search-file, and web-lookup — several independent read-only tools — in one turn, the engine batches the adjacent parallelizable ones to run concurrently; write tools are natural barriers and keep their declared order. Tool calls and their results are committed strictly in declared order, so concurrency never breaks the protocol. The most common read-only tools go from serial to parallel in one stroke, and the whole batch's wall-clock time drops noticeably.
Read-before-write + optimistic concurrency control. Before editing a file you must read it first; the engine records a baseline for the file that was read and checks the baseline hasn't drifted at edit time. When parallel workers change the same file at once, the loser gets a clear "stale" error instead of silently overwriting each other's changes. With multiple workers operating on the same workspace in parallel, this safeguard is indispensable.
Mid-run interruption folded in immediately. When a user adds another line while the agent is halfway through, the engine folds that queued message into the current turn's input at the tool-loop boundary, instead of waiting to spin it up as a separate turn. This turns "course-correct while it runs" into a natural interaction.
Loop detection. When the same tool call repeats back-to-back, the engine first nudges (on the 3rd) and then hard-stops (on the 5th); any differing signature resets the count — legitimate variations like pagination/polling won't be misfired. When the model gets stuck, it no longer silently burns tokens.
Removing the hard cap on main-turn output. Main-turn output is no longer pinned to a tiny cap, so long reports and large edits don't get silently truncated; auxiliary calls (compaction, reflection) still conservatively use a small cap.

There's one very "local" detail worth a mention: token estimation for mixed Chinese-English text. A generic estimate under-counts a pure-Chinese conversation by two or three times; the engine treats Chinese and English characters differently by character class, which is what makes the compaction threshold trustworthy. That's the kind of thing a generic SDK won't think of for you.

Taken together, these improvements answer "why not just use an off-the-shelf SDK": because the strongest capabilities of a desktop agent happen to live in the very layer an SDK doesn't expose; to make them production-grade, the loop has to be in your own hands.

2. Keeping the model always online: the multi-layered provider wrapper

The goal of the model layer is one sentence: no matter what goes wrong with a given key, a given provider, or a given network, this turn of the user's conversation should survive if at all possible. To that end, the adapter layer stacks a few wrappers on top of the engine's provider abstraction — rotation, cooldown, registration, external adaptation.

The most critical design is that the rotator sits below the runner. The engine writes the user's message into the persistent session before it ever calls a provider; if you did retry/rotation at the engine level, you'd either resubmit the user message or have to write a whole session-rollback. By placing the rotator beneath the engine, the user message is written exactly once, and "retry with another candidate" is completely transparent to session state.

The rotator's judgment is also restrained, centered on the line of the first content event:

A failure before the model emits any substantive content (text/tool call) — safe to switch to the next candidate;
Once the first content event is emitted — stop rotating and let the error propagate up, because the model may already have run a full turn, and redoing it would repeat side effects.

Error classification decides "rotate, don't rotate, or retry." Account-level failures like auth failure, insufficient balance, rate limiting, expired subscription — mark a cooldown and rotate; transient network failures like a connection reset — no cooldown, retry statelessly a few times in place; while malformed requests, content policy, and server 5xx — which would fail just the same with another key — pass straight through without rotating. The cooldown is a ten-minute, in-process, non-persistent hint: it's only a short-term signal, not worth writing to disk on every failure, and a process restart is exactly the right moment to re-probe.

On the provider "roster" side, the refactor flattens three kinds of sources into one unified abstraction:

Orkas-managed LLM: a server-side proxy, ready to use once signed in, with the server routing between text/image models;
Bring-your-own key: standard mainstream large-model providers;
External direct-connect adapters: a batch of models that need direct connection or carry their own billing, hand-adapted onto the same provider interface.

To the layers above, all of this appears as just one stable (provider, model) pair — rotation, cooldown, and external adaptation are all hidden inside the adapter layer.

3. Group chat as orchestration: from static plan-DAG to commander-in-the-loop

This is the most "rewire-your-brain" part of the refactor.

The old model was static planning: the model first generates a plan/DAG and the executor runs by the graph. The new model tears that graph out entirely and replaces it with a dynamic, commander-in-the-loop group-chat orchestration.

Its metaphor is a group-chat room:

The Commander is the room's host, not an invisible middleware;
Agent workers are first-class, equal members in the room;
All interactions are asynchronous messages, enqueued through one single message bus (there is no private path for parallel fan-out).

The commander's "dispatch" is not an @somebody written in prose — an LLM writing @AgentA in body text is just markdown from training data, and untrustworthy. The real dispatch signal is a structured tool call, and after the refactor it converges into three semantically clear actions:

dispatch_to — send an agent to run to completion and hand the result back, with the commander synthesizing. Multiple independent tasks can fan out concurrently.
run_worker — a sub-task the commander owns itself, with the result returned synchronously; an anonymous worker is the commander's "hand" (invisible to the user), while a named worker is a visible specialist.
hand_off_to — hand the conversation to the agent; the commander steps out, and the agent answers the user directly with no synthesis on top this turn.

Why group chat, rather than an orchestrator or a sub-agent tree

Making multi-agent into group chat brings several benefits a traditional orchestrator / sub-agent tree can't get:

Visibility slices. Each message is only appended to the slice of "those who can see it." When an agent worker starts up it replays only its own slice, so another agent's large output won't pollute its context. The commander sees everything.
Minimal state. The whole orchestration's core state is just "who currently holds the floor" plus a lightweight task ledger. No DAG, no complex state machine.
Naturally replayable and syncable. Messages sort naturally by timestamp, so reloading and cross-device sync both fall directly onto the message stream. The mobile end does remote control precisely off this stream — all agent compute runs on the desktop, and mobile is just a mirror render, needing no special orchestration protocol.

The team's architecture review was equally blunt on this point: a group-chat bus plus a commander-in-the-loop is Orkas's multi-agent shape; stacking another parallel sub-agent dispatch path inside the process would instead violate the invariant of "only one group-chat dispatch path."

What's new in this version: interactive hand-off

The newest piece on this line is the interactive agent hand-off.

The pain point is concrete: a "tutor"-type agent teaches the user for a turn, the user wants to keep asking follow-ups, but the system forces the floor back to the commander, leaving the user to re-@ that agent for every single line.

The solution is a server-authoritative floor + a model-decided recipient:

The floor becomes a persistent state field, saved across reloads, and rides the existing state-change event to auto-sync to every end — no new event type needed.
After the commander uses hand_off_to to give the floor to an interactive agent, the user's subsequent "no-@" messages go straight to that agent, until the agent hands back on its own or the user re-addresses the commander.
The agent hands control back with a <handback /> marker; parsing strictly verifies a true match (so a stray <handback appearing in prose isn't misread as a hand-off).
If there's an unfinished task ledger at hand-back time, the commander picks it up from the ledger and carries on.

There's also an experience fix that comes with it — commander loop bubbles. The commander's "dispatch → read result → dispatch again" loop within one turn used to be flattened into a single bubble, and on reload it would even jump out of order to the bottom. The refactor cuts one turn into multiple segments at each visible-dispatch boundary, each segment a standalone message with an ascending timestamp — for the first time the user can see the commander "looping through orchestration," and the reload order is correct too.

Finally, two safety nets that run throughout: group abort is the single stop path for all actors (the moment the user hits Stop, every worker's abort signal is cut, with even anonymous sub-workers covered by a fallback match); and the previously mentioned interrupt-steer, folding the user's mid-run interjection into the current turn.

4. From a closed catalog to an open host

If the first three through-lines were about making the foundation solid, this one is about throwing all the doors and windows open — turning Orkas from a closed catalog into an open host — while holding the security boundary without giving an inch.

The refactor systematically dismantled several "closed" choke points:

External packages. The user gives a repository address, and Orkas hosts it locally cloned verbatim into a folder — never normalized, never rewritten, never synced to the cloud (because it contains third-party dependency directories). A standalone command-line tool owns the install/update/start-stop lifecycle, scans whether it's "skill-shaped" (carries a skill-description file) or "CLI-shaped" (carries an executable entry), and writes the metadata into a registry outside the package directory (so future pull-updates never conflict). Dependency installation goes through "ask once, remember" two-step confirmation; executable entries get shims generated and injected into the bash tool's PATH, so the model can call these third-party CLIs directly.
Multi-root skill loading. The single entry point for skill execution went from recognizing only two roots to four tiers — custom / marketplace / external package / global — resolved by priority; scripts inside an external package prefer the package's own bundled dependency environment. This is the choke point with the highest regression risk, and it's backed by a full fixture matrix.
Global skill interop. Orkas reads directly from the global skill directories that other agent tools on the user's machine already maintain, achieving interop at the skill level — a skill the user accumulated in one place is usable in Orkas too. The user placing a skill into those directories is itself the authorization, so it's enabled by default, with a master switch left in. These third-party skill descriptions are an untrusted prompt-injection surface, so they go through the "open-tier" loader, are visible only to the commander, and structurally cannot enter an agent's skill allowlist.
User-configured MCP. Connectors are no longer a hard-coded catalog. The user can add any MCP server — a remote HTTP form (low risk) or a local subprocess form (high risk). The form itself is the consent surface (the command the user typed in by hand is shown verbatim), the transport config (including secrets) goes entirely into encrypted storage, and custom instances always carry a fixed prefix so they can never impersonate an official connector in the catalog.
Reverse bridge: letting the machine's external agents perceive Orkas in turn. This is the most interesting piece. External agent tools already on the user's machine used to be a black box to Orkas; now, when Orkas dispatches them, it injects a bridge channel that lets them in reverse list/read/run Orkas's skills, call connectors, and search the knowledge base. The bridge runs over a local inter-process channel (no network port opened), authenticated with a one-time credential that is unique per run and destroyed the moment the run ends. Every connector call with an external side effect goes through a user confirmation dialog — not a heuristic read/write judgment by tool name (which would err on the side of being too loose), but one confirmation per (agent, connector), with an optional "always allow."
Long-tail coding posture. The commander's decision tree gains a branch: when there's no matching agent/skill/connector, assess solving it directly with bash plus a short script, do it this turn, verify the output, and optionally offer to crystallize it into a custom skill. This comes with background bash execution (long tasks detach from the current turn, logs go to a file) and user-granted directories.

Open, but not hands-off

What you fear most when opening the doors and windows is a draft. The discipline of this refactor: not a single one of the spawn choke points for "dangerous actions" is touched. MCP starts from exactly one place, skill execution goes through exactly one runner, and bash goes through exactly one sandbox executor. On top of that, several layers of depth are stacked:

File operations always pass through the path sandbox (workspace + current attachments + directories the user explicitly granted), while credential directories, system directories, and Orkas's own directories cannot be granted;
Dangerous bash (exfiltration, destructive deletion, privilege escalation, sensitive paths) triggers a permission confirmation, with the decision split into "just this once / for this run / deny," and logs recording only category and length — never the command text;
External-package installation fail-closes and outright refuses packages with symlink members (to prevent using a symlink to read sensitive files outside the sandbox into scope), and the clone source is restricted to a protocol allowlist;
All credential-bearing transport/secrets are encrypted at rest, and bridge credentials are isolated per run;
The open-source / hosted distribution strips host-exclusive capabilities by a trimming rule.

In one sentence: every explicit user action (install / grant / submit form / click confirm) is the credential of consent, and every consent is confined to the boundary it deserves.

5. Getting smarter across sessions: memory and self-evolution

The foundation overhaul also redid two subsystems that "make the agent smarter the more it's used," both following the same engineering discipline — off by default, bounded, observable.

Cross-session memory uses hybrid retrieval: vector semantic search + keyword search (BM25), merged via RRF (reciprocal rank fusion) to avoid either single channel failing; it lands in local storage (with a full-text index). Memory comes in two kinds — the agent's own notes and the user-preference profile — each with a character cap, scanned for injection threats before writing, and frozen-injected into the system prompt at the start of every turn. The whole memory system serves only to make the agent understand the current user better; the data always stays local, and the user can view, edit, and export it anytime in settings.

Self-evolution is an agent-private skill library (stored separately from the platform's shared skill library) plus a layer of metacognitive reflection. The engine decides whether to reflect by a set of weighted signals: user correction (highest weight), recovery from a non-trivial error, task complexity, a known weakness being triggered or overcome, skill ineffectiveness... reflection only fires when the weighted signals exceed a threshold. The reflection itself is a background periodic task (roughly one round every 12 hours, a several-hour cooldown, a multi-day fallback) that uses a cheap small model to read a summary of recent activity and decide whether to create/patch a skill and update the agent's "competence profile."

The most important safety point: self-evolution is enabled only for sessions that have an agent explicitly bound — the default commander session does not evolve. Reflection has a dual token cap (count + total), one agent's failure doesn't block the others, and per-run cost is pushed extremely low. Make the agent smarter, but don't let it run away.

Engineering philosophy: earned complexity — don't simplify it away

The team ran multiple rounds of architecture review during the refactor, and one conclusion kept recurring, worth pulling out on its own: distinguish "organizational hypertrophy" from "earned complexity," and only touch the former.

The sync engine's multiple merge strategies, the self-built agent loop, the multi-layered provider wrapper, the boundary of mobile remote control — these look complex, but every layer earns its keep (multi-device eventual consistency, deep integration, multi-key rotation, a product-decided end boundary). Forcibly "simplifying" them would only lose data and muddy the layering.
What truly should be touched are the "god modules" and local duplication: extracting the stateless pure functions out of the bloated group-chat bus (prompt assembly, commander tools, the CLI turn), and collapsing the "confirmation dialog" pattern that was duplicated several times into one shared component.

What backs this kind of judgment is a set of hard disciplines written into the project's constraints document: boundary (single process, IPC as the only path, the runtime can only be dynamically loaded), layering (the dependency direction of each layer), single source of truth (categories, telemetry taxonomy, domains), and a mandatory "prompt audit" on every prompt-facing commit. What lets the foundation be overhauled without collapsing isn't some clever design — it's these invariants being held continuously.

Closing

Put the four through-lines together, and what this ground-up refactor swaps onto Orkas is an agent foundation that is self-controlled, provider-agnostic, dynamically orchestrated, open to the outside, and able to self-evolve:

A two-layer engine/adapter in-process runtime that brings a full set of coding-agent strengths — file operations, local search, system tools, parallel multi-worker, long-horizon solving — natively onto the desktop and makes each one production-grade;
A multi-layered model layer that keeps the conversation alive through key/provider/network turbulence as much as possible;
A group-chat-style, commander-in-the-loop multi-agent orchestration that swaps "static planning" for "dynamic decision," and for the first time makes hand-offs between agents feel natural;
An ecosystem moving from a closed catalog to an open host, with external packages, global skills, custom MCP, and the reverse bridge all opened up — while the spawn choke points didn't budge an inch;
And memory and self-evolution that are off by default, bounded, and observable.

Features can be added one at a time, but a foundation is only worth seriously overhauling once. Once it's done, whatever you build on top goes faster — and that's exactly the outcome this refactor was after.

Rewriting the Agent's Foundation: A Ground-Up Refactor of Orkas