Trading Hours, Live Traffic and One Very Nervous DNS Change: Inside a Friday Migration That Shouldn't Have Happened

There's a certain kind of confidence that only arrives on a Thursday afternoon, when the office is quiet, the coffee is decent and someone says the fatal words: "It'll be fine, we'll do it Friday morning."

This is the story of one of those Fridays. A two-person team at a mid-sized UK retailer — think a few thousand orders a day, a WooCommerce front end bolted onto an aging on-prem server in a cupboard in Leeds — decided to move the whole thing to a cloud provider. During trading hours. With a marketing email scheduled for 4pm.

What follows is a reconstruction, hour by hour, of what actually happened. Names changed, egos bruised. Consider it a disaster-avoidance guide wearing a war story's clothes.

08:15 — The Optimism Phase

The plan, such as it was, looked like this: spin up a new cloud instance, rsync the site across, do a database dump and restore, flip DNS, done by lunch. The dev who'd written the plan — let's call him Dave — had tested it thoroughly.

On a staging copy.

Of a site from three months ago.

The first wobble came early: the on-prem box was running an Ubuntu release so old that the cloud provider's migration tooling simply shrugged at it. Nothing insurmountable, but it ate forty minutes nobody had budgeted. Lesson one: your migration plan should start from what's actually running, not what you believe is running. Run cat /etc/os-release before you write the runbook, not after.

10:30 — The Database Dump Nobody Waited For

The database was around 18GB — mostly order history and product images in longtext fields, because of course it was. The [mysqldump](https://en.wikipedia.org/wiki/MySQL) they'd timed on staging took eleven minutes. On the production box, shared with the live site and a cron job that hadn't been told to stop, it took fifty.

Worse, the dump locked enough tables that checkout slowed to a crawl. Two abandoned baskets appeared in the logs. Not catastrophic, but the first real signal that "during trading hours" was a bet, not a plan.

If you must migrate live — and sometimes you must — schedule your dump for your lowest-traffic window and use --single-transaction on InnoDB. Dave knew this. Dave had used it on staging. Staging had nothing else hitting the database.

12:45 — The Backup That Was Never Tested

Here's the bit that should make every reader sit up. The team's rollback plan was, and I quote from the runbook: "restore from backup if needed."

There was no backup. There was a dump, taken that morning, which had never been restored anywhere to verify it wasn't corrupt. When Dave's colleague — the sensible one, let's call her Priya — asked when the restore had last been tested, the honest answer was: never. Not once in the four years the box had been running.

This is the single most common failure mode in small UK firms. Backups exist. Restores do not. A backup you've never restored is a hope, not a contingency. Test your restore path on a scratch machine, on a schedule, with a calendar reminder. It takes an hour a quarter. This story nearly took a business with it.

14:20 — DNS, Caching and the Longest TTL of Someone's Life

The site was synced. The new instance passed smoke tests. Time to flip DNS.

Except the zone's TTL was set to 86400 seconds — a full day — by an agency that managed the domain in 2019 and hadn't touched it since. Flipping the A record meant that for up to 24 hours, some customers would hit the old server and some the new. With a card processor session tied to the domain, that's a recipe for orders vanishing mid-payment.

They dropped the TTL to 300 and waited. And waited. Resolvers across the UK — your BT, your Virgin, your office Pi-hole — held onto the old record with varying degrees of stubbornness. The migration became a slow-motion split-brain: two servers, both live, neither authoritative.

Lesson: check your TTL a week before you need it low, not an hour before. Lower it early, let the old value expire, then migrate.

15:50 — The Card Processor Quirk

By mid-afternoon the new site was serving traffic and everything looked golden until the first real order failed at the payment step. Then the second.

The issue: the payment gateway whitelisted the old server's static IP. The cloud instance had a different egress IP, and every request was being bounced. The fix was a ten-minute phone call to the processor's support line — but it was 3:55pm on a Friday, the marketing email had just fired, and the queue was "unusually busy."

Twelve minutes of dead checkout during a promotional send. Not a disaster, but it cost real revenue and rather more in blood pressure. Whitelist your new IPs before cutover, not after. It's a five-minute job when nothing's on fire.

17:30 — The Rollback That Wasn't Needed, Barely

By early evening, orders were flowing, the old box was idling, and Priya was updating the incident doc. The rollback plan — restore that unverified dump — was never invoked. Had it been, they'd have discovered the dump was truncated. They found out the next week, when they finally tested it, as a matter of routine rather than panic. It failed.

The Honest Scorecard

The migration worked. Eventually. But the postmortem wrote itself:

• Inventory reality, not intention. Check OS versions, cron jobs and TTLs before planning.
• Migrate in a quiet window, even if "during trading hours" sounds braver in the retro.
• Test your restore path quarterly. An untested backup is decorative.
• Pre-register new IPs with your payment processor and any third-party API that whitelists.
• Write the runbook for the person having the worst day of their year, not for the version of you sitting calmly on a Thursday.

Dave and Priya got there in the end. The site's been on the cloud for eighteen months now, faster and cheaper. But the runbook for the next migration starts with a single line, learned the hard way: "Assume nothing is as it seems until you've checked it yourself — including the backups, especially the backups."