Building a local AI assistant that operates your machine

Most AI assistants live in the cloud: your conversation goes to a provider, and the assistant can only act on a small, predetermined set of surfaces. A local assistant is the opposite proposition — it runs on your machine and works with the tools you already use, which means your data, your models and your confirmation loop.

This is the design behind Kestro, a local AI assistant that can operate the browser, files, terminal and connected tools like Telegram. Here's how I think about building one.

The core: surfaces, not skills

The cleanest mental model is to treat every capability — browser, files, terminal, Telegram — as a surface the assistant can act on through a small, explicit interface.

• Browser — navigate, read pages, identify elements.
• Files — read, write, move, rename.
• Terminal — run commands and scripts.
• Telegram — send and receive messages on your behalf.

A surface has typed actions and explicit permissions. The assistant doesn't "know" how to control everything on your computer; it knows how to make well-defined calls against these interfaces. That boundary is what keeps the project maintainable and the behavior predictable.

One provider layer, many models

Model providers change their APIs regularly, so the assistant should not know or care which provider answered. A provider layer wraps Claude, GPT, Gemini, Grok, LM Studio and Ollama behind one interface:

assistant <- tool loop <- provider layer <- { claude, gpt, gemini, grok, lm-studio, ollama }

Two selection modes cover the real workflows:

• Manual — you pin the model for a session or a task. Deterministic, transparent, useful when you need a specific behavior.
• Automatic — the assistant picks the best configured model for the job. Convenient, but slightly less predictable.

The safety loop is structural

A local assistant that can write files and run commands is powerful, and power without control is just a liability. Three mechanisms keep it in check:

1. 01Confirmation gate. Read actions run quietly; write and destructive actions pause and ask. The gate is part of the flow, not an afterthought — so it can't be skipped by default.
2. 02Audit log. Every action — requested, approved and executed — is recorded. When something goes wrong, the log tells you what happened instead of leaving you guessing.
3. 03Explicit scope. The assistant operates browser, files, terminal and Telegram. It is not a general controller of every native desktop application. Keeping scope honest is a feature, not a limitation to apologise for.

What to build first

If you're building your own, start with one surface and a text-only model backend. Get the confirmation + audit loop right before adding anything else — the loop is the product. Then add a second surface, then a third, then local models.

The payoff

A local assistant is slower to build than a chat wrapper, but the trade is real: your data stays on your machine, you can use local models when privacy matters, and every destructive action passes through your own approval. For an independent developer, that last part is the whole point — you're teaching the computer to work with you, not instead of you.