How to Rotate API Keys After a Breach Without Downtime (A Runbook)

Updated November 2, 2026, with Northflank's October 30 breach and what it told customers to rotate. Updated October 28, 2026, with a section on vendor breaches and third-party scripts after the Amplitude incident. Updated October 24, 2026, with Railway's preliminary incident report, a no-downtime rotation order and an audit-log checklist.

Knowing how to rotate API keys used to be a once-a-year chore. This week it's urgent for a lot of people. On October 21, Railway said an unauthorized party may have accessed customers' encrypted environment variables and decrypted some of them. The same morning, a poisoned LiteLLM 1.107.0 was published to PyPI and Docker with a credential stealer inside.

Different incidents, same job: assume the secrets are gone, replace them without taking the app down, and make sure the old ones stop working. This post is the runbook. If you want the prevention side for PHP dependencies, that's in my Composer supply chain checklist.

Breach notice in your inbox and no time for a runbook? Get a free quote and I'll help you work through it.

On October 22, Railway published a preliminary report on the incident. Here's what it confirms, in Railway's own terms:

• Scope. Railway says at most 1.85% of all workspaces had some environment variables potentially compromised, about 100,000 workspaces. It's reaching out to customers with definite impact, plus a larger group it considers high risk.
• What the attacker wanted. Railway says the attacker appeared to be after AI provider and platform API keys. It sees two higher-risk groups: workspaces with a very large number of environment variables, and environments with a lot of AI vendor keys.
• What wasn't touched. Railway says individual user accounts weren't logged into and running services stayed online the whole time. It isn't ruling out access to user source code yet and says it will confirm in its final report.
• The window. Railway puts the attack between 8:58 AM CDT on October 20 and 10:32 PM CDT on October 21 (13:58 UTC October 20 to 03:32 UTC October 22).
• Vendor revocations. Railway worked with GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe and others to revoke potentially exposed keys. For some vendors it sent every key stored on the platform. Railway says a revocation email from a vendor doesn't by itself mean your key was exposed.
• What Railway reset. Railway revoked all Railway access tokens and rolled every GitHub App installation. If your GitHub email differs from your Railway email, you may need to log in with "Log in using email."

What Railway tells customers to do: if you signed up before October 21, rotate the sensitive variables and keys you store on Railway. Remove public network access to your databases. Check your GitHub audit log and review every app with access. Check the audit logs in your container registries, such as Docker Hub.

A vendor revoking a key for you isn't the same as you rotating it. If a provider killed your old key, your app may already be failing. Generate a new one, deploy it, and then work through the rest of this list.

Railway. If you keep environment variables in Railway, Railway recommends that every user roll all of them as a precaution, including API keys and private registry credentials, and review logs for unusual activity. It says it will email the customers it confirms were affected, but don't wait for that email. Railway also reset two things for you: the S3 credentials for every storage bucket and the GitHub tokens it held. Your services need a redeploy to pick up the new bucket keys, and you have to reconnect GitHub yourself. GitHub tokens that were stored as registry credentials keep working on GitHub until you revoke them there. Railway paused deployments during the investigation, so check its status page before you plan a redeploy.

LiteLLM. You're affected if you ran an unpinned pip install between 11:39 and 13:23 UTC on October 21 (6:39 to 8:23 AM Central). Docker counts too, if you started the v1.107.0, latest, main-latest or main-stable images between 11:46 and 14:20 UTC (6:46 to 9:20 AM Central). The payload ran every time Python started, even if nothing imported LiteLLM. It went after cloud credentials, SSH keys, environment variables, AI tool configs and the provider keys stored in the LiteLLM proxy's database.

For LiteLLM, clean up before you rotate. LiteLLM says not to start with pip, because any Python start runs the payload again. Delete cook.pth, cook.py and sitecustomize.py from site-packages, kill the background Python process, then rebuild from scratch pinned to litellm==1.106.2. Rotating keys on a box that's still infected just hands over the new keys.

Not every breach this week is about your keys. On October 22, Capcom disclosed unauthorized access to CAPCOM ID affecting 11,774,247 accounts. Email addresses, login timestamps and last-login IP addresses were exposed, and 43 of those records also included birth date, gender, country and nickname. Capcom says it found no compromise of user-set passwords, but someone changed the passwords on 196 accounts, which have since been reset. No credit card data was involved.

The bigger risk now is phishing that pretends to be Capcom. If your work email is tied to a gaming account like this, expect convincing fake "reset your password" emails, and never rotate a key from a link in one.

On October 28, Coinbase said a security incident at Amplitude, an analytics vendor, "impacted a small subset of Coinbase customer contact data, among other Amplitude customers." In Coinbase Support's statement, the company says it found no evidence of unauthorized access to its own systems, customer funds weren't affected, and it's notifying affected customers and regulators. It hasn't said how many people were affected, which fields were exposed, or when it happened. A figure of more than 70,000 users is going around on X, but Coinbase hasn't confirmed it. When I wrote this, I couldn't find a public statement from Amplitude.

The same day, three South Korean financial firms reported breaches. Daishin Securities said personal and credit information for 38,490 customers leaked from a server tied to opening investment accounts online. The data included names, email and postal addresses, and encrypted account and mobile phone numbers. The crypto exchange Bithumb said an outsider got into its internal employee messaging and work app and took personal information. It says that app is separate from trading and is still checking whether member data was involved, though Yonhap, citing an industry source, reports that some customer information leaked. The asset manager Mirae Asset Global Investments said an outside IP address viewed some internal business systems, including some employees' names, and that investor data wasn't leaked.

The Coinbase case is the one most small businesses should learn from, because your customers' data doesn't only live in your own systems. Every analytics tag, chat widget, payment SDK and CRM integration you add holds either a token or a copy of that data. Here's how I handle that side of things:

1. Inventory every third-party script and SDK. Check your tag manager, your page source, your WordPress plugins, package.json and composer.json, and any mobile app SDKs. For each one, write down the vendor, what it collects and who owns the account.
2. Scope and rotate vendor tokens. Give each vendor its own key with the narrowest permissions it offers, such as read-only access, one project or an IP allowlist. When a vendor reports an incident, rotate the keys you gave it and any keys it uses to call your systems, using the same no-downtime order above.
3. Send analytics less. Don't send email addresses, phone numbers or names as event properties or user IDs. Use an internal ID instead. Turn on IP and PII masking where the tool supports it, and stop collecting fields you never look at.
4. Review vendor access. List the OAuth apps connected to Google Workspace or Microsoft 365, GitHub, Slack and your CRM. Remove old vendors and integrations you no longer use, and make sure vendor accounts use SSO or MFA.
5. Get ahead of the phishing. Leaked contact data is what phishing runs on. Tell your customers how you'll actually contact them, and that you'll never ask for a password, a one-time code or a payment from a link.

If you're not sure which scripts and vendors touch your customer data, get a free quote and I'll help you map them and lock down the tokens.

The trick is never having a moment where the only valid key is the old one, or where the app only knows the new one before the provider accepts it. I call it a dual-accept window: for a short time, both keys work. Here's the order I work in.

1. List every secret the system could read. That means environment variables, .env files, CI secrets, registry credentials, cloud roles, database passwords, OAuth client secrets, webhook signing secrets, SSH keys and AI provider keys. If a process on that host could read it, it goes on the list.
2. Rank by damage. Cloud and admin credentials come first, then anything that leads to more secrets (GitHub tokens, registry logins), then databases, then anything that moves money or sends email and texts as you. AI provider keys can run up a bill fast, so they come right after.
3. Create the new key next to the old one. Most providers let two keys be active at once, or let you set the old key to expire later. Check your provider's docs before you start. If a provider only allows one key, plan a short maintenance window for that one.
4. Store the new value in one place. Update the shared variable or secret manager, not five pasted copies.
5. Redeploy everything that reads it. Containers read environment variables at startup, so a new value does nothing until each service restarts. Go one service at a time, and watch error rates after each.
6. Confirm traffic is on the new key. Most provider dashboards show when each key was last used. When the old key's "last used" time stops moving, you're ready.
7. Revoke the old key today. Not "once things settle down." An old key that still works is the whole problem.
8. Prove it's dead. Make one request with the old key and expect a 401 or 403. I don't count a key as rotated until I've seen that.

Start with anything that leads to more secrets. A cloud role or a GitHub token can read other credentials, so an attacker with one gets the rest. After that, rotate the database password, because it's the one that leads straight to customer data.

Some secrets aren't a simple swap.

• Database passwords. Create a new database user or password, point the app at it, redeploy, then drop the old login. If you change the password in place, every running copy of the app fails until it restarts.
• Webhook signing secrets. The sender signs with the secret and your app verifies it. Have your app accept both the old and new secret during the switch, then flip the sender, then remove the old one from your code.
• JWT signing keys. Publish the new public key first, then start signing with the new private key. Keep the old public key around until every token signed with it has expired. A kid (key ID) header makes this painless.
• OAuth client secrets. Some providers let you add a second client secret before you delete the first. Others don't, so check before you start.
• Registry and deploy tokens. Railway notes that GitHub tokens stored as registry credentials keep working on GitHub until you revoke them on GitHub.

The key you forget is the one that gets used. These are the usual places:

• A pasted copy in a second service instead of a shared reference. Railway's own guide calls this out.
• A local .env on your laptop, or a teammate's.
• CI variables and deploy scripts.
• A cron job on a different server that nobody remembers setting up.
• AI coding tools. The LiteLLM stealer specifically went after editor and agent config folders, including ~/.cursor.

Rotation stops the next use of a key. It doesn't tell you whether someone already used it. For the window your host says was exposed, check:

• GitHub: your account and organization audit logs, plus the list of authorized OAuth and GitHub Apps. Railway calls this out specifically.
• Container registries: Docker Hub or whichever registry you use. Look for pushes or pulls you don't recognize.
• Cloud accounts: audit trails for logins, new keys or new users you didn't create.
• AI providers: usage dashboards for spikes, unfamiliar models or traffic at odd hours.
• Payment and messaging providers: API request logs for anything you didn't send.

Write down what you rotated, when, and what the logs showed. If a client or your insurer asks later, you'll want that record.

Rotation can't un-leak data. If the stealer had the database password, the data may already be gone. Rotation stops the next use.

Short outages happen. Even with overlap, some service will be holding the old value. Do it in a quiet window if you have the luxury, but don't wait days for one.

Your vendor's revocation isn't your revocation. Railway says it told upstream AI providers to revoke keys proactively. Check yours anyway.

The work is tedious, and that's the point. Every step you skip is a key that still works for someone else.

Each secret lives in one place per environment, and everything else points to it. Each environment gets its own keys, so a staging leak doesn't touch production. AI keys get spending limits where the provider supports them. Agents get the narrowest access I can give them, the same rule I use in my AI coding setup. If you moved a Salesforce integration to OAuth, like I covered in the SOAP login() retirement post, the client secret goes on the list too.

Most of the integrations I build connect two or three outside services, so each one comes with its own rotation plan. You can see what that work looks like on my integrations page.

How do I rotate API keys safely?

Create the new key, update it where the secret is stored, redeploy every service that reads it, then revoke the old key. Finish by making one request with the old key to confirm it now fails.

Do I need to rotate keys if Railway didn't email me?

Railway recommends that all users roll every environment variable stored in Railway as a precaution. Its email goes to customers it has confirmed were affected, but the precaution applies to everyone.

Is LiteLLM safe to use now?

LiteLLM says only 1.107.0 was compromised, and it has been pulled from PyPI, Docker Hub and GHCR. It recommends pinning litellm==1.106.2 or another explicit earlier version, and rebuilding any environment that ever ran 1.107.0.

What's the difference between rotating and revoking a key?

Rotating replaces a key with a new one. Revoking makes the old key stop working. A rotation isn't finished until the old key is revoked.

Can I rotate API keys without downtime?

Usually, yes. Create the new key while the old one still works, deploy it everywhere, confirm traffic has moved, then revoke the old key. The only real downtime risk is a provider that allows just one active key at a time.

How many Railway workspaces were affected?

Railway's preliminary report says at most 1.85% of all workspaces, about 100,000, had some environment variables potentially compromised. Railway still recommends that everyone who signed up before October 21 rotate the sensitive variables they stored there.

A vendor revoked my key. Was my key leaked?

Not necessarily. Railway says it sent some vendors every key it stored, regardless of whether those keys were known to be affected, so a revocation notice alone doesn't mean your key was exposed. Replace the key either way, because the old one won't work anymore.

Do I need to rotate keys if one of my vendors was breached?

Rotate any keys you gave that vendor and any keys it uses to reach your systems, even if the vendor says only contact data leaked. If customer contact details were exposed, the bigger risk is phishing, so warn your customers about it too.

I'm on Northflank. Do I need to rotate my environment variables?

Yes. Northflank told every customer to assume all environment variables were decrypted and to rotate all of them, including secrets on workloads, secret groups and global secrets, even if you haven't seen anything suspicious. Start with AI provider keys, then cloud, payment and database credentials. Revoke each old key with the company that issued it, not just in Northflank, sign in again to reconnect your integrations, then check your AI usage and billing for calls you didn't make.

• Railway, "Security Disclosure: Wednesday October 21st"
• Railway, "Preliminary Report: Oct 21st Security Incident"
• Railway, "Security Disclosure Recommendation: Redeploy Buckets and Reconnect GitHub After the Credential Reset"
• Railway status page
• LiteLLM, "Contained: Oct 21, 2026 LiteLLM release v1.107.0 Compromised"
• OSV, "MAL-2026-2144: Malicious code in litellm (PyPI)"
• Capcom, "Notice and Apology Regarding Customer Information Affected Through Unauthorized Access" (October 22, 2026)
• Coinbase Support, statement on the Amplitude incident (X, October 28, 2026)
• CoinDesk, "Coinbase says user funds are safe after vendor exposes customer data"
• Yonhap, "(LEAD) South Korean financial firms confirm data breach of customer data" (October 28, 2026)
• Seoul Economic Daily, "Mirae Asset Global Investments Hit by Hacking, Says No Client ID Data Leaked"
• CoinDesk, "Bithumb hacked through employee system as South Korean financial firms face cyberattacks"
• Polymarket Status, incident update (October 29, 2026)
• DebugBear on X, security incident notice (October 29, 2026)
• CNBC, "Some Polymarket user funds stolen, matter referred to UK authorities" (October 29, 2026)
• Hudson Rock (InfoStealers), "Polymarket Breach Alert: Wallet-Draining Code Found in a Third-Party Script" (October 29, 2026)
• MDN, Subresource Integrity
• MDN, Content Security Policy (CSP)
• BTCPay Server blog, "Security Incident Affecting BTCPay Server 2.4.1 and Earlier" (November 2, 2026)
• BTCPay Server on X, hot wallet sweep alert (November 1, 2026)
• BTCPay Server on X, incident update: attack vector targeting 2.4.1 and older (November 1, 2026)
• Nicolas Dorier on X, attacker logged in as admin (November 1, 2026)
• Nicolas Dorier on X, keep hot wallets small (November 1, 2026)
• Andrew Camilleri (Kukks) on X, updating after the fact is not enough (November 1, 2026)
• Andrew Camilleri (Kukks) on X, attacker scanning for instances (November 2, 2026)
• BTCPay Server blog, "Security Advisory: Update BTCPay Server to 2.4.2 Immediately" (August 7, 2026)
• BTCPay Server 2.4.2 release notes, GitHub (August 7, 2026)
• BTCPay Server 2.4.7 release notes, GitHub (October 29, 2026)
• Bitquery, "BTCPay Server Hack: $1.6M Swept From Merchant Hot Wallets" (November 2, 2026)
• BTCPay Server docs, User Guide (wallet setup)
• btcpayserver-docker, btcpay-down.sh
• Northflank status page, "Security incident" (October 30, 2026, updated November 2)
• Northflank, "How to migrate off of Railway" (October 22, 2026, with an October 30 edit note)
• Wiz, "Black October" for AI Developer Platforms (October 30, 2026)
• Blockaid on X, community alert on flap.sh and kintsu.xyz (October 30, 2026)
• Blockaid on X, ten more sites serving the Step Drainer (October 30, 2026)
• Flap on X, DNS hijacking incident (October 30, 2026)
• Kintsu on X, DNS hijack at its registrar (October 30, 2026)
• Kintsu on X, incident update (October 31, 2026)
• Kintsu on X, domain registered through Vercel Domains (October 30, 2026)
• Kintsu on X, site restored, root cause still being confirmed (October 31, 2026)
• Kintsu on X, frontend takeover, not a contract exploit (October 31, 2026)
• ICANN, EPP Status Codes: What Do They Mean, and Why Should I Know?
• CISA, Emergency Directive 19-01: Mitigate DNS Infrastructure Tampering
• Let's Encrypt, Certificate Authority Authorization (CAA)
• Cloudflare Docs, CAA records
• ICANN, DNSSEC: What Is It and Why Is It Important?
• Cloudflare Docs, DNSSEC
• Vercel Community, Enabling DNSSEC for domains purchased through Vercel
• crt.sh, Certificate Transparency search
• Revoke.cash