Build your own background agent!
I recently shut down my startup, Engines. Coming off that experience, I felt a huge burst of creative energy to build and explore.
I had some inklings that AI coding interfaces didn’t align with my sensibilities (mainly achieving flow while coding with agents) and I want to build my own background agent interface to see how it would work.
Here’s my experience building a background agent on top of the Claude Code SDK with Claude Code itself.
First I needed to choose a sandbox provider. Initially I started with Daytona but I struggled to get Claude running. It turns out that even though the Claude Code SDK runs headless, the underlying libraries used still require a terminal present. This posed an issue on Daytona but eventually I was able to figure out a workaround.
Once I had chosen my sandbox provider, I chose a hosting provider. I’ve used Render in the past and liked it so it was the obvious choice.
I decided to build the backend in Rust and the frontend in Typescript. Mainly because I’ve been enjoying using Rust for AI projects, since I find the compiler can catch many potential errors before they are committed.
Building something end to end that worked took a week or so. Building something reliable has taken longer.
The first roadblock I ran into was that Daytona did not support SSH from outside the sandbox. That was initially a critical requirement because I wanted to support SSH into the container so I could test things locally.
So I switched to Modal. Surprisingly this was very straightforward since Claude had written a decent Rust interface for the sandboxes.
Once I switched to Modal, I was enamored with how convenient it was and how clean its interface is. It’s incredibly fast. The startup times for its containers are incredible. Less than 10 ms due to two clever proprietary filesystems.
I was able to quickly build a custom image for my background agents to have the right compile time tools.
The most useful learning by far in this process has been to understand what background agents are useful for.
I have come to see background agents as one of three things:
- Exploratory agents for a new feature
- Quick bug fixers when I know the root cause or can point to a set of potential causes
- Executors for concrete plans
I have come to the conclusion that for these tasks, the agent itself doesn’t matter much. I switch often between Claude Code and Codex and I find the results typically are about the same.
That’s surprising! I would expect Opus to yield dramatically better results but I don’t find that to be the case.
My take on the future is that background agents will become a commodity. Everyone will have one for the aforementioned tasks above and you will likely rely on a competent but not SOTA model to get things done.
Since I was programming almost entirely with background agents, I came up with another tool to speed up planning. I found myself relying often on Repo Prompt (link) but I found the copy/paste tedious.
I wanted to select files for the context window just like I would in Zed, with a Crtl + P workflow.
So I built Swarm Desktop. It’s a chat interface that allows you to drop files into GPT 5 and chat with them. GPT 5’s context window is quite large so depending on your codebase, you can drop plenty of files.
You can then ask for detailed plans based on your desired changes and get back a plan that Claude Code or Codex can follow.
Swarm Desktop also allows you to spawn a task to Swarm Cloud with the click of a button.
I find myself using this often when I discuss a plan at length with GPT 5 and then have it output the final plan for implementation.
Through this all, I have found my preferred way to work with agents and I’ve built more of an intuition for when these tools are useful and when they need more direction.