~/blog · · Andy Rodriguez

Deploys that resume themselves

Why an interrupted deploy on SetCloud picks up where it left off instead of failing outright — checkpointed phases, orphan claiming, and billing that never double-charges.

A deploy pipeline is a sequence of slow, external, and occasionally flaky steps: fetch source, build an image, push it, roll it out, wait for health checks to pass. Any one of those steps can be interrupted by something that has nothing to do with your code — a platform-side process restart, a routine maintenance replacement, a transient network blip talking to a build service. The naive response to an interruption partway through is to mark the whole run failed and make you retry it from scratch, paying again for whatever already completed. SetCloud’s deploy pipeline doesn’t do that. It resumes.

Every phase checkpoints

The orchestrator that drives a deploy runs through the same phases every time — source, build, push, service update, stabilization — and after each one finishes, it persists that fact to the run’s own record before moving to the next phase. That persisted state is the whole trick: a run isn’t a single long-lived process holding everything in memory, it’s a sequence of durable checkpoints, any one of which can be the starting point for picking the run back up.

Progress during a run streams live too — a per-run log list that multiple watching clients (the dashboard, an open CLI session) can tail simultaneously — but the log stream is a side effect of the run, not the source of truth. The source of truth is the checkpointed phase state, because that’s what survives a restart of the process actually running the deploy.

Claiming an orphaned deploy

If the platform-side process driving a run disappears mid-phase — a task replacement, a crash, anything that isn’t your code failing — the run doesn’t just sit there forever showing “in progress.” A background reconciler checks periodically for runs whose heartbeat has gone stale: no progress reported for longer than a run at that phase should reasonably take. When it finds one, it claims the run and fires off a resume rather than waiting for a human to notice a stuck deploy in the dashboard.

Resuming isn’t a full restart from a blank slate. The reconciler reconstructs everything the run needs — which repository, which branch, which credentials to use for the build — from the database rather than from in-memory request context, because the process that originally held that context is the one that’s gone. Once reconstructed, the run re-enters the same orchestrator any live deploy uses, landing back on the phase where its last checkpoint left off.

That claiming process isn’t unbounded, deliberately. A run that keeps failing to make progress after being resumed a few times over stops getting auto-resumed and is instead marked as a terminal failure you can see and act on — a deploy stuck in a genuine failure loop shouldn’t retry forever silently.

Idempotent billing

The part that matters most for your invoice: resuming a run never re-runs billing for work that already happened. If a build already completed and produced an image, and the interruption happened afterward — during rollout, say — resuming the run recognizes that the build phase is done (the checkpoint says so) and skips straight past it, including the billing event tied to it. You’re charged once for the build that ran once, no matter how many times the platform had to pick the deploy back up to get it fully live.

Build settlement uses the deployment run as its billing identity. A resumed run reuses completed build billing. Additional running-resource time or a genuinely new build can still add costs.

What you see

From the dashboard, most of this is invisible by design — a deploy that got interrupted and resumed just looks like a deploy that took a little longer than usual, with the same Logs tab showing the same phases in the same order. If you want to see the mechanism, watch the Logs tab during a long deploy: build output, image push, health checks, in phase order, same as any run. A build failure caused by your own code — a broken script, a missing dependency — still fails outright and never replaces your previously running version; that distinction is deliberate. What’s automatic is recovery from platform-side interruptions, not recovery from bugs in what you pushed.

More detail on reading a deploy’s logs and telling a resumed run apart from a genuinely failed one: deployment logs and troubleshooting. First deploy walkthrough: deploy an app from GitHub. Product overview: /products/deployments.