Setting up my own data building factory with Herdr and Linear
I took the idea of a software factory and built a small one for my own data platform work: a chief of staff, a PM per product, pods of agents on a VPS, and Linear in the middle.
In this article
Software factories are everywhere at the moment. I keep coming back to a simpler question, though: what does a factory look like for those of us who build data platforms?
The article that made the idea feel real for me was Jacob's piece on how Spawn is built. By his telling, a swarm of agents now develops the product, at a token bill of more than $3 million a month. The agents pick up and triage their own issues and commit straight to main. If there's a conflict or a bug, it's theirs to figure out. There are constraints to keep production stable, but no pull requests in the usual sense. I'm not sure how much of that I'd want to copy. It's thought-provoking, but it sits at the far end of the spectrum.
The framing I found more useful came from Tessl. In Context-Driven Factories, Dru Knox describes a progression from skills, to loops, to factory. After more than ten years working with data platform owners, I have a fairly good idea of the skills and loops involved in building a data platform. So I wanted to see what it would take to slowly introduce a factory into the development of my own.
Prefer to watch? Here's the full walkthrough:
Skills, loops, factory
The Tessl progression is simple, and that's why I like it.
Skills are what we already know how to do. In our case, that's writing models, working with Dagster, setting up infrastructure. It's also the less visible craft: what a release actually is, and how to cut it into tasks someone can pick up. Tessl's version is to write those workflows and standards down as plain files, run them by hand and refine them until you trust the output. Most of us are already doing a version of this. We have our plugins, and vendors now ship plugins that package skills for their own tools.
Loops are skills that run on their own. One example: once the business requirements are written, a loop turns them into technical requirements and documents them in a Linear issue. A loop improves as it runs, and those improvements carry to every place the skill is used.
A factory is what you get when enough loops join up. Work moves from initial requirements all the way to delivery with agents picking it up at each step, with a human involved where it matters, but otherwise as autonomous as it can be.
The loops already exist in data platform work
For data work, I think the interesting part is how many loops there are once you look past engineering.

I built this map a long time ago, for my own reference on client work. We talk a lot about the engineering side: ingesting sources, development cycles, DevOps and CI/CD. But a data platform also needs a roadmap, releases cut from that roadmap, progress tracking, business stakeholders brought into the work, a working definition of quality and someone looking after the assets. Each of those boxes involves a skill. Automate one and it becomes a loop.
So a factory run for a data platform could start with the business. You work out the roadmap with them, cut the roadmap into releases and shape each release into a brief. From that brief onward, everything could be automated. That's the vision, anyway.
My own skills live in Lab, the plugin where I keep my workflows, playbooks and skills. It's quite out of date and needs time I haven't given it. But workflows and skills are the two pieces I lean on every day, and they're what the factory runs on.
The factory floor
What I've built so far is a first-level implementation. It runs in my own environment, across more than one machine, but it isn't scaled to an organization.

I'm the control room. From my Mac, I use Herdr as the multiplexer for all the terminals my agents live in. Some of those terminals are local. Others run on a VPS, so the pods don't burn through my own machine. I can also follow along from my phone, though that's the least interesting part.
The roles are where it becomes a factory rather than a pile of terminals:
| Role | What it does | Reports to |
|---|---|---|
| Chief of staff | My daily conversation. Triages everything coming back from the PMs and relays only what needs my attention or a decision | Me |
| PM (one per product) | Owns the roadmap, triages issues in Linear, shapes the work and dispatches it to a pod | Chief of staff |
| Pod lead | Spawned for one issue. Reads it and decides which roles it needs: engineer, designer, architect, security, QA | PM |
| Pod members | Do the work in their own worktree and hand back to the lead | Pod lead |
The chief of staff wasn't in the original design. I added it once I had several PMs working side by side, all spinning up pods, and everything flowing back to me at once. It was just too much. I needed a role whose job is to triage all of it.
Linear sits in the middle of every interaction. Initiatives are products, projects are releases and issues are what pods do. The diagram also shows the guardrails I set up front: no more than two pods in flight, a daily spend cap and an inspection gate that checks CI, a preview, the report and the Linear update before anything reaches me.
One loop, end to end
To show it working, I asked the chief of staff for the smallest possible change: add one line to the README of the factory repo.

The chief filed the request in Linear and passed it to the factory PM. The PM shaped the issue and spun up a pod. The pod lead decided an engineer was enough for a one-line change, the engineer made the edit and opened a pull request, and the lead handed it back.

The PM closed the pod, which was no longer needed, and reported back through the desk. A few seconds later, the chief of staff told me I had a pull request to review.

The PR had the single line I asked for. That's a trivial change, and I picked it on purpose. In an earlier dry run of the same flow, it took about 48 seconds from the PM reading the issue to a gated PR. What I wanted to see was the shape of the loop: request, shape, pod, PR, gate, back to me, with me deciding only at the two ends.
Herdr does most of the plumbing
This is simpler than it sounds because Herdr already handles agent-to-agent communication. It describes itself as an automation layer for coding agents, in which one agent can create work for other agents, check their state and collect their results.
The recipes in the docs show the pattern: start a named reviewer agent in a specific pane, with a chosen kind of agent and model, prompt it, wait for it to settle and read its output. Herdr also connects to other machines, which is how the pods on my VPS show up next to my local panes.
So the factory isn't much more than roles written as files, Linear as the shared record and Herdr passing messages between panes. To me, that's the encouraging part. The Spawn setup is on the far side of the spectrum, but the fundamental ideas are simple, and you can set up a version for your own work.
Scaling this beyond one person
I'm the only human on this plant floor. Scaling it to an organization is a different problem: how a whole team interacts with a fleet of agents, and how all of that effort gets coordinated.
The best treatment of that I've seen is Zach Lloyd's talk, What it takes to build a software factory. Warp didn't invent the idea, but I think they've done a lot to make it popular, and they sell their own factory platform. The talk is obviously about their product too. Still, it covers the whole stack and is generous with ideas on how humans and agents should interact, beyond what any one product does. It's worth a listen.
For data platform owners, I'd expect the range of approaches to be wide, from prudent to aggressive. My setup sits deliberately near the prudent end: one human gate, small pods and a spending cap.
Where I'd start
If you're a data builder curious about this, I wouldn't start with agents. I'd start by writing down your own release cycle, from requirements to delivery, and the skills and loops involved in it. A map like the one above is enough. Once you can see the loops, picking the first one to automate gets much easier.
I'm curious how other data builders are setting up their own factories. Our space is a bit different from software in general, but I think we can borrow most of these ideas.