The Rolling Handoff

The app I work on is a crowdfunding-style platform. People land mid-donation, card details half typed, a progress bar creeping toward a goal. It runs on one modest VPS: no Kubernetes cluster, no fleet of load balancers, just three services (a public app, an admin console, and a background worker) sitting behind a reverse proxy. For most of the project's life, deploying it meant running docker compose up -d: stop the old container, start the new one. Most nights that gap is a second, maybe two, and nobody's around to notice. But this app doesn't really get quiet nights. Someone is almost always mid form, and a one second gap lands as a failed request at the exact moment they were about to finish giving money.

I didn't want to fix that by adding servers. I wanted to fix it by changing how one server hands traffic from the old version to the new one.

The shift

The fix wasn't more hardware, it was a different mode. The same box now runs Docker Swarm instead of plain Compose, and deploys go out with docker stack deploy instead of docker compose up -d. Swarm's routing mesh keeps the published ports reachable the whole time an update is happening, so whatever's forwarding public traffic to those ports never has to change at all. What actually changes is how a new task gets in before the old one leaves, and that turned out to hinge on four separate, unglamorous settings all agreeing with each other.

1. Start the new task before stopping the old one. order: start-first on every service tells Swarm to launch the new task alongside the running one and only stop the old task once the new one has passed its own healthcheck. The default order is the opposite: stop, then start. That's exactly the gap I was trying to get rid of.

2. Every deploy needs a name Swarm has never seen before. Swarm's change detection keys off the image reference, not off a "please redeploy" command. Ship the same tag twice and nothing happens. So every deploy targets an immutable, per commit tag. The mutable latest tag still gets pushed too, but that's for a human running docker images later, not for Swarm to act on.

3. A rollout that fails puts itself back. With failure_action: rollback set, if the new task never turns healthy, Swarm reverts to the previous one on its own. Nobody has to be awake at 3am to notice a bad push and undo it.

4. The pipeline checks the orchestrator's homework. After Swarm reports the update applied, the deploy workflow still polls docker service ls until every service reads 1/1, then curls both health endpoints from the box itself. "Applied" and "actually healthy" aren't the same claim, and I'd rather a rollout that quietly never converges fail the CI job loudly than sit broken in silence.

Picture the overlap window this creates. Requests keep flowing across the top with no gap at all. The old task keeps serving while the new one starts underneath it, gets polled on /health a few times, and only once it reports healthy does the old task start draining and finally stop. There's no moment where neither task is up. The two lanes overlap on purpose.

# docker-compose.yml: per-service deploy block
deploy:
  replicas: 1
  update_config:
    order: start-first
    failure_action: rollback
    delay: 10s
  rollback_config:
    order: stop-first
  # healthcheck lives on the image itself, see below

One detail cost me an actual debugging session before it clicked. The Dockerfile HEALTHCHECK for each service originally shelled out to curl, and curl gets purged from the runtime image right after an earlier install step to keep the image small. The healthcheck was silently failing against a binary that no longer existed. All three services now ship a five line Node script as their healthcheck instead. No new dependency, and it runs on the same runtime already sitting in the container.

# deploy workflow: convergence + health poll
# after `docker stack deploy` returns
until [ "$(docker service ls --format '{{.Replicas}}' -f name=web)" = "1/1" ]; do
  sleep 5
done
# repeat for admin + worker, then confirm from the box itself
curl -f http://127.0.0.1:3010/api/health
curl -f http://127.0.0.1:3011/api/health

"Applied" is a claim the orchestrator makes about itself. "Healthy" is the only claim worth trusting.

None of the four mechanisms above are exotic, and that's kind of the point. Every one of them is a config flag or a five line poll loop. What actually earns the zero downtime claim isn't the switch to Swarm itself. It's refusing to trust the orchestrator's own success message until something outside it confirms the same thing twice.

What I still don't know until it survives the real box

I wrote all of this before running it against production traffic for the first time, and I'd rather flag what I couldn't verify from a laptop than pretend a design doc is the same thing as a working system.

• Swarm's advertise address. Initializing Swarm mode needs an --advertise-addr, and the workflow auto detects it from the default route with a plain fallback. If both guesses are wrong, that's a one time manual docker swarm init on the box, not a pipeline problem.
• Service discovery inside the overlay network. The database hostname resolved fine under plain Compose's bridge network. Whether it resolves the same way once Swarm's overlay network is involved is untested against the real host. If it doesn't, the failure mode is a clearly failing migration step rather than a silent one, but it's still a real unknown.
• Registry auth on the box. The deploy assumes the VPS's Docker daemon is already authenticated to the image registry, since a manual pull already worked there before. Nothing new required, just a dependency worth naming instead of assuming.

Writing that list down before the first deploy, instead of discovering it live, turned out to be the more useful habit, honestly, more useful than any of the Swarm config itself. The config makes a bad rollout revert automatically. The list is what makes sure I'd actually notice if something upstream of the config was wrong in the first place.

Written from a working deploy pipeline for a small crowdfunding platform. Specifics of the app itself are intentionally left out. The mechanism is the point.