Long-running AI agents: I was wrong about the gimmick

A few months ago I had a firm opinion about long-running AI agents. I thought they were a gimmick.

The idea sounded silly to me. You give an agent a task, walk away, and come back much later. What you get is a confident pile of code that half works. The longer it runs, the more it invents. Hours of drift, no human in the loop, and at the end a diff you can't trust.

That was my view. I said it out loud, to people whose opinion I respect.

I was wrong.

Today I almost always have two or three agent sessions running long. Not for minutes. For days. I run several harnesses side by side — Claude Code, Codex CLI, OpenCode, Hermes Agent — on real work. Client projects. Internal tools. The content pipeline that will publish this post.

This week I opened my session list and one card stopped me. It had been running for 4 days and 17 hours. It had spent $172.16. It had added 3383 lines and removed 371. The context window was 84% full. The title I gave it, months ago, half as a joke: the new "norm".

TL;DR: Long-running AI agents stopped being a gimmick for me. The models now keep the goal across hundreds of steps, and a good harness gives them memory and real tools to work with. I keep two or three sessions running for days. I check in three or four times a day and review the plan and the diff instead of writing every line.

One of my running sessions. The title was a joke when I typed it. It isn't one anymore.

"You just let it run for four days, Jack?" — you'll ask me. And it's a great question. No, I don't just let it run. I check in. I read the plan. I read the diff. I correct the course and let it go again. Four days of wall-clock time is maybe twenty minutes of my attention, spread thin. So what actually changed? Four things.

1. The models hold the thread now

The old failure was memory. The agent would forget the goal by step forty and start solving a different problem. That got better fast. The models I use now keep the intent of a long task in their head across hundreds of steps. They still make mistakes. But they make them while working on the right thing, which is a very different kind of mistake.

This is not only my impression. METR, a research group that tests AI agents, measures tasks by how long they take a human expert. Then it checks which of those tasks an agent can finish. In March 2025 they reported that this task length had doubled about every 7 months for six years. Their number describes how hard a task is. It says nothing about how long the agent runs. But it points the same way as my session list.

We felt it in our own tools too. We deprecated the first two Magus plugins, GTD and Kanban, because the models grew up and stopped needing them. I wrote about that in Kanban is for people. The agent outgrew it.

2. The harness gives the agent a desk, not just a chat box

This is the part people miss. The model is only half of it. The other half is the harness around it — the thing that gives the agent a place to work.

A good harness gives the agent persistent memory, so a fact from Monday is still there on Thursday. It gives it skills it can load for a specific job. It gives it scheduled jobs and real tool access — the file system, the shell, the browser, my repos. The agent stops being a clever text box. It becomes a worker with a desk and a set of tools it knows how to use.

If you want a shorter definition, Anthropic's engineers wrote a good one. The harness is the loop that calls the model and routes its tool calls to the right place. The model decides what to do next. The harness does it and keeps the notes.

3. Small check-ins beat one giant prompt

I used to write one enormous prompt and hope. That is the wrong shape. A four-day task does not survive one perfect instruction.

What works is steering. I give a short brief, let it run, read what it did, and nudge. "Wrong file." "Split that." "Stop, the test is lying to you." Each check-in is small. Three or four of them over a day keep a long session on the rails better than any wall of text I could write up front.

Anthropic's team saw the same thing from the other side. In their long-running experiments, the agent tried to build the whole app at once. It ran out of context in the middle of a feature and left the next session to guess what had happened. Their fix was simple. They asked for one feature per session, with a git commit and a progress note at the end of each one.

4. My job moved from writing to reviewing

Here is the honest bit. I write less code than I did a year ago. My hands are on the plan and the diff now, not on every line.

A developer reads the diff, not the promise. That sentence is the whole job now. I judge plans before they run and I judge changes after. The agent does the typing. I decide if the typing was any good. It's a smaller job in keystrokes and a bigger one in judgment.

Where it still falls apart

None of this makes the caveats go away. They are real, and I want to name them, because a post that only sells you the good part is a brochure.

It fails when you delegate badly. A vague brief running for four days produces four days of confident garbage. The agent won't stop and ask if the whole direction is wrong. It will happily build the wrong thing with great energy. The model said it was 95% confident. So was I, before I checked. The quality of the output is the quality of your brief, multiplied by time.

The bill is real. $172.16 is not nothing for one session. That's about $37 a day. Run three of those a week and you are paying for a junior salary in tokens. It is worth it when the work is worth it. It is a waste when you point it at a task you hadn't thought through.

And 84% context is not a badge. It means this session is near its end. The window is almost full, and once it's full the agent starts forgetting the early decisions. So I plan for it. I have the session write a summary, hand off to a fresh one, and carry the state across. A four-day run is not one long thought. It's a relay, and you have to be ready to pass the baton before the runner drops it.

Anthropic's engineers use almost the same picture. They compare a long task to a project staffed by engineers working in shifts, where each new engineer arrives with no memory of the last shift. Their answer looks like mine. A progress file and the git history, so the next session knows where the last one stopped.

How I start a long-running AI agent session

If you want to try this, here is the short version of what works for me. Nothing here is new. It's the four changes above, turned into steps.

1. Think the task through before you start. A vague brief running for four days produces four days of confident garbage.
2. Run it inside a proper harness. The agent needs persistent memory and real tool access: the file system, the shell, the browser, your repo. If you start with Claude Code, read Boris Cherny's Claude Code tips first.
3. Start with a short brief and let it run.
4. Check in three or four times a day. Read the plan, read the diff, and nudge.
5. Watch the bill and the context. When the window gets close to full, ask for a summary and hand off to a fresh session.

So no, the gimmick I dismissed didn't quietly become good. The gimmick was never the point. The harness got better. The models got better at staying on task. And I changed my mind, out loud, the same way I stated the wrong opinion in the first place.

Two or three sessions, always running. That's just how I work now.

FAQ

What is an agent harness?

It's the software around the model that gives it a place to work. A good harness adds persistent memory, skills, scheduled jobs and real tool access: the file system, the shell, the browser and your repos. Claude Code, Codex CLI, OpenCode and Hermes Agent are all harnesses.

Can AI agents really run for days?

Yes, with check-ins. The session in this post ran for 4 days and 17 hours. I checked in three or four times a day to read the plan and the diff and correct the course.

How much does a long-running agent session cost?

The session in this post cost $172.16, about $37 a day. It's worth it when the work is worth it. On a task you haven't thought through, it's a waste.

What happens when the context window fills up?

The agent starts forgetting the early decisions. So before the window is full, I have the session write a summary and hand off to a fresh one that carries the state across.

Do I still need to review the code?

Yes. A developer reads the diff, not the promise. The agent does the typing, and I decide if the typing was any good.

These jokes were created by AI. Please double-check before laughing.