/bruno
All writing

Self-hostingAI Agents6 min read

Our Coding Agents Outgrew GitHub's Runners

A hand-drawn bar chart made of paper stacks labelled Jun to Oct, each taller than the last. A robot labelled agent climbs a ladder to put PR #694 on the October stack while another robot carries a box of more PRs. On the right, a small server labelled €99 a month, with six of its eight lights on, is sweating, and a stick-figure reviewer stares up at the tower and says 694?!
One stack per month. October needed a ladder.1-bit / hover to resolvehold to resolve

In the last week of October, our company opened 694 pull requests. In June it was between 42 and 75 a week. Nobody here got ten times faster at typing: most of those PRs were written by coding agents and opened under our names. Mine went from 4 PRs in two weeks of June to 241 in that one week of October. (Sorry, reviewers.)

Every one of those PRs runs CI, and every CI minute on GitHub's own runners is billed. So in the middle of October we moved almost all of it to one rented server for €99 a month, where a tool called GARM gives every job its own fresh virtual machine.

Here's what the agents did to our CI, why GARM, and how the first three weeks went.

What the Agents Did to Our CI

For most of the summer, CI was background noise: 2,500 to 8,000 GitHub minutes a month, and a bill of $15 to $47.

Then, on 22 September, Anthropic released Claude Opus 5.5, and for us it was a game changer. That week we opened 458 pull requests, up from 244 the week before, and GitHub billed us for 19,930 minutes in seven days, more than twice as much as in all of August. It never came back down: 475, 476, 620, 633 and 694 pull requests a week through October. In my own agent sessions since then, 99% of the model calls went to Opus 5.5 or its smaller sibling, Sonnet 5.5, which came out six days later.

Two hand-drawn bar charts on the same June-to-October axis. Top: pull requests per week, from 42 in early June to 458 in the week Opus 5.5 came out on 22 September and 694 in the last week of October. Bottom: minutes on GitHub's runners per week, peaking at 19,930 that same week and falling to 611 after our own server took over on 14 October.
The top kept growing. The bottom stopped once the server took over.

(To be fair, the first big day was the Monday before the launch, when a colleague's agents opened 65 pull requests overnight on one project. Every week after the launch was bigger than that one, though.)

September ended at 35,870 minutes and $216. October came to 40,001 minutes and $242, and that's with most of its second half already moved off GitHub.

Most of that jump came from one client project. In the week of 21 September it got 348 pull requests, up from 21 two weeks earlier, and in the month to mid-October it used 60% of all our CI time. When I asked where that time came from, the answer was a bit embarrassing: 79% of it was triggered from my account. My agents, my bill :P

The money alone wouldn't have moved us. $242 a month is fine. The rest hurt more:

  • Slow jobs. GitHub's standard runner for private repositories has 2 CPUs. Our biggest Flutter job took 23.4 minutes there, at the median.
  • A ceiling. The Team plan runs at most 60 jobs at once. We hit it, and the busiest moment in those 30 days had 70 jobs going across all our runners.
  • No quiet hours. Even overnight, one or two jobs were running on average, from agent runs and scheduled jobs.

Picking the Server

We looked at a few alternative runner providers first. Then I asked an agent (of course) to pull our Actions history: twelve months of billing, and every one of the 26,200 jobs since mid-July. Its minute count matched GitHub's bill to 99.4%, so I trusted the rest, and the rest was a surprise. In the 30 days before, nothing at all was running 63.5% of the time, and on average 1.4 jobs were. We were paying for bursts, not for load.

Its first suggestion was a 16-core machine with 128 GB of RAM. I asked it to leave out the one project's surge, aim for €75–100 a month, and let jobs queue. Replayed through a simulated queue, our last 30 days fit on one much smaller box, as long as each job got fast cores.

The next morning we rented a dedicated server for €99 a month, and it was online 44 minutes after the order went in.

Why GARM

The plan was never just "install GitHub's runner on a server". That part is easy, but then every job shares one machine: whatever the last job left on disk, whatever it installed, and a single Docker daemon. Two jobs that both start Postgres on port 5432 collide, and a docker system prune in one job deletes another job's containers. Our workflows were written for ubuntu-latest, which is a clean machine every single time, and I wanted to keep that promise.

GitHub's own tool for autoscaling runners is actions-runner-controller, which is a Kubernetes operator. We don't run Kubernetes anywhere, and I wasn't going to start for this.

GARM fit what we already had. It's one Go binary with a SQLite file inside. It listens to GitHub's job queue, and for every job it asks Incus for a fresh virtual machine from our image, registers a runner for exactly that one job, and deletes the machine when the job is done. It polls GitHub for work, so GitHub never has to reach our server.

A hand-drawn diagram. GARM on our server asks GitHub for jobs, gets job 1 and starts a fresh VM just for that job, and the VM goes in the bin when the job is done. Six more VMs sit booted and waiting, and the next job takes one.
Every job gets a clean machine, and nothing survives it.

A fresh VM per job also changed who gets root. With a plain runner, every workflow in our 40-odd repositories can reach the host's Docker socket, which is root on the box. With GARM, a job is root inside a VM that's deleted a few minutes later.

Fresh machines take time to boot, so we keep six of them booted and idle. On day one, with a single warm VM, a job waited a median of 58 seconds for a runner. With six, it was 3 seconds.

GARM has its limits. It's young (we run v0.2.1), and it tells you very little about what it's doing: there are no job metrics, so we wrote a small collector that records every job for Grafana. (The same records drive the office TV, where every running job is a little car on a road. Another post.) It also only drives Incus for us, so when two old MacBook Airs joined the pool for iOS builds, they got their own small scheduler. Also another post ^^

The First Three Weeks

Our job log only keeps two weeks. From 19 October to 2 November, our runners did:

  • 5,440 jobs from 40 repositories, 328 hours of job time.
  • A median wait of 3 seconds for a runner. One job in ten waited more than about 5 minutes, and the slowest 1% about 25.
  • About one job in eight started between 10 pm and 7 am.

And on GitHub's side:

Week ofPull requestsMinutes on GitHub's runners
5 October47619,359
12 October (move starts on the 14th)62012,977
19 October6334,049
26 October694611

The server costs €99 a month. GitHub Actions now has a $20 spending limit with a hard stop, so the agents can open as many PRs as they like and the bill stays flat. They just queue.

Shortfalls

  • Busy afternoons are slow. All eight VMs share one CPU, and the same job can take two to three times as long as on a quiet morning. An agent first blamed the CPU's power limit, then measured and corrected itself: the limit costs about 25%, and most of the rest is the sharing.
  • It's one box. If it's down, CI is down. Thanks to the $20 limit, that includes the deploy that would fix it, so that one runs from my laptop.
  • GitHub could still charge for it. In December 2025 GitHub announced a $0.002 per minute charge for self-hosted runners, then put it on hold. Our October bill has no such line. If it ever comes back, our current volume would add about $84 a month, and CI would cost more again every time the agents get busier.

If Your Pull Requests Are Climbing Too

Look at where your CI minutes go before you look at servers. GitHub's usage report breaks them down by repository and day, and ours said we were paying for bursts. A box that's idle most of the day handles bursts fine, as long as nobody minds a short queue, and agents don't mind at all.

694 pull requests in the last week of October, 611 minutes on GitHub's runners. The agents haven't noticed a thing.

Until next time <3

End of transmission