Leaving a breached host in one morning

On 21 October 2026 the platform that hosted our game's server published a security notice: an attacker may have read the environment variables it stored for customers, API keys included, and its deploys had been paused since early morning. We chose to treat every secret it held for us as read. The next morning we moved the server, its database and its secrets to another cloud behind a front door on a name we own, shipped new client builds that name it, and merged the rows players had written in between. The new builds went live two and a quarter hours after "go"; the old projects were deleted a little over six hours after it.

This is that day as a runbook for a small studio that runs an AI game master on a hosted platform. The checklist comes first. The sections after it say what each line cost us. Times are our local time (UTC-6).

Problem solved. Your hosting provider is breached and its deploys are paused, so you can neither trust the secrets it holds nor rotate them in place. This is the order of operations we used to move an AI game's server, database and secrets to another cloud in one morning, including the client builds that named the host.

How to use it. Tonight, the first seven lines of the checklist. Tomorrow, the rest. Each line's reason is in the sections below. Seven lines are marked core: if you have time for nothing else, do those.

The checklist

Tonight. Nothing a player can reach changes tonight.

• ☐ Core. Decide stay or leave on two numbers: how many hours deploys have been paused with no time to recovery posted, and whether the attacker's stated target includes the issuer of your model key. Ours were twenty hours and yes. If both are bad, leave: a provider that cannot redeploy cannot help you rotate.
• ☐ Core. Cap the model key's spend tonight, in the provider's console and in your own server's daily ceiling. It is the asset most worth stealing from a game like ours.
• ☐ Core. Inventory every secret the host holds, by name only. Columns: issuer, what revoking does, when it may be revoked. Expect to find one that exists only on the host and one you thought was there and is not.
• ☐ Core. Mint replacements into your own vault; revoke nothing yet. Revoking first takes the game down with no way to redeploy. Keys that regenerate-and-kill (a store's Web API key, a chat webhook) wait for the moment of the swap.
• ☐ Read your usage before you touch anything: your own ledger and the model provider's console, per day, written down. That is the baseline any later anomaly is judged against.
• ☐ Plan, have someone try to break the plan, then write the runbook: an owner, a done-when and a rollback per step; the human's clicks marked as clicks; every command in the shell the human uses; no secret on any command line, because cloud CLIs log their flag values to disk.
• ☐ Never carry a secret from the breached host to the new one. Every value on the new host is born there.

Tomorrow.

• ☐ Core. Build the foundation by script: project, managed Postgres (point-in-time recovery on, deletion protection on), an image registry, and keyless deploy from CI (a workload identity, not a stored key). Set the billing project on every command.
• ☐ Secrets by pipe into the new secret store, each verified by one live call that proves the value without printing it (the model list endpoint; a ticket check). Label every version with where it came from.
• ☐ Restore the newest dump before the first boot, as the application user, with exact row counts asserted, and every sequence floored high (we used 100,000,000) so the old host's last rows merge later with their own ids.
• ☐ Deploy by digest. /health reports the build commit; /ready reports the database. CI asserts that prod is running the commit it built.
• ☐ A front door on a name you own: TLS at the edge, bot checks off for the API host (a challenge page reaches a Steam player as "couldn't confirm access"), an allowlist of the paths your client calls, and a proxy timeout longer than your longest request.
• ☐ Core. Gate 1: the new host refuses correctly. A real client build gets 401 on a bad ticket and 400 on a malformed request. Refusal is the proof the path is whole; a 200 proves less.
• ☐ Core. Gate 2: one real turn, read from the new host's own request log. Set nothing live before it.
• ☐ Only if your client ships a hardcoded host: cut and ship builds that name your own hostname and send a User-Agent naming the build; post the "restart once" note before the old host stops.
• ☐ Final dump of the old host, then the delta merge: dry run, apply, and a second dry run that must refuse because the ids are now present.
• ☐ Inventory everything that still names the old host (scheduled tasks, CI secrets, probes, runbooks, the client's error paths) and switch or retire each. Delete the old projects. Keep the account until every archive is verified, then delete it before the next bill.
• ☐ The week after: read the new host's backup list on day one; rewrite any local backup job; grep every runbook and the client for the old provider's name, not just the hostname; re-verify every rotation still marked "pending" with a probe that tells the two keys apart.

What happened

Railway's incident opened at 01:24 and paused builds and deploys for the day. At 15:59 its status page added a security notice, and its disclosure post that evening (blog.railway.com/p/security-disclosure-wednesday-october-21st) said an unauthorized third party "may have accessed encrypted environment variables" and for some "obtained the decrypted contents", tied it to attacks "with the aim of compromising AI inference providers", and asked customers to "immediately roll all environment variables, including any API keys". Deploys stayed paused.

For a game server that is a short, specific list: the key that bills every narration, the key that verifies a player's store ticket, the secret that signs session tokens, the secret that seeds the server-side dice, the admin token, the alert webhook, and the database URL for the Postgres that holds every exchange a player has had with the game master. We do not know what was read; we chose to treat all of it as read.

Two facts shaped the night. You cannot rotate in place while deploys are paused: a new key cannot reach the running service, and revoking the old one first takes the game down for every player with no way back. And the hostname was baked into every shipped build, so leaving the host meant shipping a client release.

Stay or leave

Stay and rotate: mint every replacement now, swap them in at the first redeploy the provider allows, revoke the old values last. Or leave: stand up the same server somewhere else, born with new secrets, and never redeploy on the breached host at all.

At 21:18, twenty hours into the outage with no time to recovery posted, we decided: "railway being down for 20 hours now without any eta with a full compromise, we're done there." Three forks were settled in the next ten minutes. The server would run on Google Cloud (Cloud Run for the container, Cloud SQL for Postgres, Secret Manager for the secrets) because we could stand all three up by script before morning. Cloudflare would front it, so that the next move is a DNS change and not a client release. Every new build would name api.isentroth.com, a name we own; the old host had been the provider's own subdomain, which is why leaving meant shipping.

The rotation plan was not wasted. Its inventory and its order (mint first, revoke last) became the first half of the migration. The difference is that the new values were born on the new host instead of swapped into the old one.

The night: inventory, mint, baseline, plan

The inventory, by name, never by value. Writing the table produced three findings we would not have had by memory: the dice-seed secret existed only on the host (nowhere in our own vault), so it had to be minted, not copied; two CI read tokens we assumed lived on the host were CI secrets and owed no rotation; and the private container image meant the host also held a registry credential outside its variables list.

Mint first, revoke last. The replacements went into our vault that evening: a second model key in the same billing workspace, a new signing secret, a new admin token, the dice-seed secret. Every old value stayed live. The store's Web API key and the chat webhook were left for the morning, because regenerating them kills the old one at once. The old model key stayed live until the old host's turns read zero at 14:00 the next day.

A read for misuse, not a guess. Our own ledger's last 24 hours and the provider console's per-day spend for the key both showed the usual shape for the week, with no burst. Both were written down before anything changed, so a later anomaly would have a baseline.

Two plans, two adversarial reviews, one runbook. A second reader was asked to break the migration plan; the corrections were applied and the result reviewed again. The reviews removed an organisation-policy change, replaced a load balancer with a plain edge Worker at a fraction of the cost, set the edge timeout above the server's own longest request, dropped an edge rate-limit rule because a 429 reads to a player as "you have hit today's limit", and moved the database restore to before the first boot. The runbook had twelve steps, an owner per step, a done-when per step, and a rollback per step that read "leave it" until the step where players move.

The morning in twelve steps

The part that is not about hosting

A web platform changes its host with a DNS record. A Steam game cannot, if the host's name is a constant in the client. Ours was, and the only override the client accepted was a loopback address for local testing. Every live build (demo, playtest, and the retail build sitting in the store's review queue) named the breached host.

• The cut is on the critical path. The build, the two gates and the set-live taps were a third of the morning.
• The rollback after set-live is a last resort. It is either the edge pointing back at the old host, which works only while the keys you consider compromised are still live, or the previous build ids set live again.
• Old builds break the day the old host stops. A player who had not restarted the store client saw "the hosted game master is down" until the update applied. Post the "restart once" note before the old host is removed.
• Own the hostname, and name the build in the User-Agent. The next host move is one edge variable and no client release, and the edge log says which build each request came from. That is how we knew the old host's traffic had reached zero before deleting it.

What bit us, so it does not bite you

Eleven things, none of them in the runbook.

1. The CLI was billing a different project. Config and application-default credentials pointed their quota elsewhere, so every call answered SERVICE_DISABLED until the scripts set the target project on each command. Set it explicitly in every script.
2. A paste command assumed the clipboard. The restore step read the dump's decryption key from the clipboard. A command that depends on the buffer fails silently. It became a masked prompt, then a vault CLI call.
3. The custom domain did not save. The first dashboard add left authoritative DNS answering NXDOMAIN. Read the name from outside after every dashboard click; a green tick is not a resolution.
4. A dependency bump broke the release gate mid-cut. A certificate update had vendored its test package, which the build's content gate refused. Fixed during the cut. A bump on the day of a move is a risk you are choosing.
5. Scanners found the new origin in minutes and drew 26 500s on paths no client calls. Put the edge allowlist up before the cut, not after.
6. A rotation we thought was pending was our own mislabelled secret. After rotating the store key we probed the "old" one for eleven days and asked the store to revoke it; the store's reply showed we had been probing a different live key, because one version of the new secret had been filled from the wrong vault project. Label every secret version with its source, and verify a rotation with a probe that tells the two keys apart, not one that only says "alive".
7. An empty value is a value. A terminal without the vault binding set a read token to an empty string, and CI went red on "invalid token". The script that sets it now refuses anything that is not a token.
8. The local backup job was host-specific. It ran against nothing after the move and was disabled; the local dump went 65 hours stale before the script was rewritten. The new host's own daily backups had run every day, but nobody read that list until day three. Read it on day one.
9. The client's outage screen keyed on a header only the old host sent. It still did a week later. Grep the client for the old provider's name, not just the hostname.
10. The incident runbook named the old host 39 times six days after the move; the rollback it described no longer existed. Grep every runbook on the day.
11. Some steps are a human's, and the plan should say which. Three actions on the breached host could only be done by the account holder in its dashboard (a service switch and a credential read among them). Mark those steps as clicks, with the person's name beside them, so the morning does not stall on a refusal.

What it cost

• Time. An evening for the inventory, the minting, the plan and its reviews, and a little over six hours the next day from "go" at 08:58 to the old projects deleted at 15:15. One pair of hands ran the runbook; mine were on the dashboards, the vault and the phone. Then eleven days on a support ticket that turned out to be item 6.
• Money. Price the new stack the night before, so the decision is not made blind. Ours was planned at a small multiple of the old host's fee, most of it the managed database; the first bill has not been read yet. The edge is on a free plan. No model calls were spent on the move.
• Players. No stored turn was lost: the last row on the old host was written at 08:52, the first on the new host at 11:31, and the rows in between were merged that afternoon with zero deletions. Players on old builds saw an outage message until they restarted.

Limits

• One studio, one server, one morning, under a demo's festival-week load, not a launched platform's. A platform with thousands of concurrent players has a different migration.
• The target was Google Cloud and Cloudflare because we had used both before. The order of operations does not depend on them; the glossed terms (secret store, managed Postgres, edge proxy, keyless CI deploy) have an equivalent on every major cloud.
• We do not know whether the breach reached our data. The database had no public endpoint for two months before the incident. Whether a possibly-read database URL is a player-data incident was mine to decide and record; this note does not make it for you.
• Two roles ran the day: one on the keyboard with the runbook, one on the dashboards, the vault and the phone. One person doing both should expect a longer day.
• Every number here is from our own records of the day (the issue thread, the session log, the provider's public posts), re-read on 2 November 2026.

Where the files are: the author publishes the checklist and the runbook template at github.com/schedule3wizards/host-exit. Three scripts the note describes (a secrets sync that never prints a value, a database restore with sequence floors, an edge Worker that forwards to an origin with a path allowlist) are not published yet; the author says they will be under MIT. Arcanum has not run or audited any code.

Submitted to the Arcanum Community Library by Matt and published here with the author's permission. The note and the repository's text are shared under CC BY 4.0: share and adapt it freely, crediting "Matt, Isentroth". Arcanum has not tested or audited these claims.