Railway's October 21 Outage Stopped Our Deploys: The Rotate-Then-Migrate Plan We Used

Key Points:

• Rotate first, migrate second, and keep the two tracks separate. Rotation is a same-day job and migration is a two-week plan. Re-issue every secret that sat in Railway before October 21, 2026 (database passwords, JWT and session signing keys, third-party and AI provider API keys, OAuth client secrets, webhook secrets, Git tokens, registry credentials) straight into your own cloud secret manager.
• Use the dual-credential pattern: create, deploy, verify, revoke. Revoking the old credential before the new one is deployed is the fastest way to turn a two-hour rotation into a self-inflicted outage.
• You do not have to rebuild your apps. Railway already turns your code into container images with Railpack, Nixpacks, or your own Dockerfile. Commit a Dockerfile per service and the same image runs on Kubernetes in your own AWS, GCP, Azure, or Scaleway account - or an existing Kubernetes cluster you already run.
• The migration work lives in four places, none of them application code: databases and volumes, environment variables and secrets, private networking (*.railway.internal becomes Kubernetes service DNS), and DNS plus TLS cutover. A 5-10 service app with two databases takes days, not months.
• Cut over at DNS. Run both platforms in parallel, replicate Postgres with logical replication, drop the DNS TTL to 60 seconds 48 hours ahead, flip, and keep Railway warm for 72 hours as the rollback path.
• The structural fix is owning the account. When the cluster, the databases, and the secrets live under your own cloud identity, a platform vendor incident pauses new deploys without putting your credentials in someone else's control plane.

If your deploys stopped on October 21 and someone in the incident channel just typed "rotate everything", this post is for you. Here is the short answer: rotate today, migrate on a plan, and no, you do not have to rebuild your apps to leave Railway. Your services are already container images. What needs work is the config, the data, and the networking around them.

I am going to give you the exact rotation order, a Railway-to-Kubernetes mapping table, a 12-day parallel-run migration plan, and an honest cost comparison. This is the sequence I would run myself, and the one we use when teams move production onto a cloud account they own.

What happened in Railway's October 21 outage, and why are customers being told to rotate credentials?

Railway disclosed a security incident on October 21, 2026, in which an attacker gained "unauthorized access to a subset of internal Railway systems" and decrypted a subset of customer environment variables, so Railway recommends that "all customers who signed up prior to October 21 rotate their keys stored on Railway" (Railway preliminary report). Any secret held in a shared control plane during an incident like this is treated as exposed until proven otherwise, because rotating is cheaper and faster than proving non-exposure.

Here is what Railway itself published, and nothing more:

• Timeline: Railway spotted privilege escalation in its build infrastructure on October 20 and halted builds. The dashboard, API, and CLI went down on October 21, and Railway invalidated sessions and sent rotation notices that evening. Pro and Enterprise builds came back early on October 22, with full API and deploys restored later that day (all times in the preliminary report).
• Scope: Railway says "at most 1.85% of all workspaces had some environment variables potentially compromised", and that envelope encryption prevents it from identifying exactly which keys were affected.
• Running services: "all existing user deployments remained online and running" while new builds and deploys were paused.
• Recovery: as of October 29, new signups were still paused while Railway hardens its systems (Railway status page).
• Still open: Railway has not published the exploit details, and the forensic investigation is ongoing.

Railway also recommends removing public database access, reviewing GitHub audit logs and app access, and checking container registry audit logs. Treat that list as the floor.

The structural lesson is about blast radius. A fully managed PaaS control plane holds your build pipeline, your environment variables, and your database connection strings at the same time, so one incident touches deploys, secrets, and data access together. Credentials are the attacker's favorite door: credential abuse was the top initial access vector in 22% of breaches, and third-party involvement doubled to 30% (Verizon DBIR 2025). The average breach still took 241 days to identify and contain and cost $4.44 million (IBM Cost of a Data Breach 2025).

To be fair to Railway, everyone has bad days. One year earlier, AWS us-east-1 had a DynamoDB DNS failure that started at 11:48 PM PDT on October 19, 2025, with impact across services lasting until 2:20 PM PDT on October 20 (AWS post-event summary). Nobody gets a platform that never fails. The question is who holds your secrets and your control plane when it does.

So run two tracks. Rotation happens today. Migration happens on a plan.

What should I rotate first after a PaaS security incident, and in what order?

Rotate in dependency order: database credentials, then session and JWT signing keys, then third-party API keys, then OAuth client secrets, then inbound webhook secrets, then Git deploy keys and tokens, then registry credentials, then long-lived cloud IAM keys. Issue every new value directly into your own cloud secret manager (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Scaleway Secret Manager, or Kubernetes Secrets synced by External Secrets Operator) instead of pasting it back into the platform you might be leaving.

Start with an inventory. Export every service's variables, shared variables, reference variables, build-time arguments, and connection strings, then add every secret copied into CI, local .env files, or another tool outside Railway.

The 8-step rotation checklist:

1. Database users and passwords: create a new user with the same grants, deploy it, then drop the old user.
2. Session and JWT signing keys: rotating these logs every user out, so schedule the change and tell customers before you do it.
3. Third-party API keys: Stripe, SendGrid, Twilio, and AI provider keys (Railway names Anthropic, OpenAI, and OpenRouter among the vendors it worked with to revoke keys).
4. OAuth client secrets: issue a new client secret in each identity provider and deploy it before deleting the old one.
5. Inbound webhook signing secrets: roll the secret at the sender and accept both values during the overlap.
6. Git deploy keys and personal access tokens: replace them with fine-grained, expiring tokens, and reinstall or re-scope the Railway GitHub App.
7. Container registry credentials: rotate push and pull tokens and check registry audit logs for unexpected pushes.
8. Long-lived cloud IAM access keys: replace them with short-lived, role-based credentials wherever your cloud supports it.

The dual-credential pattern: create new, deploy, verify, revoke old. Create the new credential while the old one still works, deploy it everywhere it is used, confirm in the audit log that traffic now uses the new one, and only then revoke the old one. Never revoke-then-deploy on production, because every service still holding the old value fails the moment you click revoke.

Verify after you revoke. Check AWS CloudTrail, GCP Cloud Audit Logs, or Azure Activity Log for the old key ID, look at Stripe and GitHub security logs, and scan database login history for the old user.

Then sweep the places teams forget: secrets hardcoded in Dockerfiles, committed .env files, values baked into image layers, Slack and ticket threads, CI logs, and laptops. GitGuardian found 23,770,171 new hardcoded secrets pushed to public GitHub repositories in 2024 alone (GitGuardian State of Secrets Sprawl 2025).

Some things rotation cannot fix. Data that left already stays gone, so customer PII exposure goes to your legal and disclosure process. Anything signed with an old key (sessions, API tokens, signed URLs) stays valid until it expires or you invalidate it explicitly.

Do I have to rebuild or rewrite my apps to move off Railway?

No. Railway already builds your code into container images, using Railpack by default (Railway build docs), Nixpacks on older services, or your own Dockerfile (Railway Dockerfile docs). Moving off Railway means recreating configuration, data, and networking around those images, not touching application code.

If a service has no Dockerfile, write one or point the destination at a buildpack builder. The runtime contract stays the same: a listening PORT, environment variables, and a start command.

The container image, PORT, environment variables, health checks, replica count, CPU and memory limits, and cron schedules all carry over 1:1. Plan explicit rework for RAILWAY_* variables, service.railway.internal hostnames, reference variables, volumes, and anything that relied on Railway's build cache.

For a 5-10 service app with two databases, this is a days-not-months job when the destination already handles the cluster, ingress, TLS, and CI wiring. The long pole is data replication and third-party allowlists, never the code.

The honest hard part is that everything implicit on a PaaS becomes explicit on a raw cloud account: private networking, TLS renewal, egress IPs, secret storage, and cluster upgrades. That explicitness is the whole point, but something or someone has to own it.

What are my options for where production runs after Railway, and how do they compare?

You have four realistic destinations after Railway: another managed PaaS, raw cloud with Terraform and a Kubernetes stack you build yourself, managed Kubernetes operated by your own team, or an internal developer platform installed into your own cloud account. The deciding difference is who owns the cloud account, where secrets sit at rest, and what a vendor control-plane incident can reach.

1. Another managed PaaS (Render, Fly.io, Heroku, or staying on Railway): the fastest move with the lowest effort, and the same structural exposure, because your workloads and secrets still live in someone else's account.
2. Raw cloud, Terraform, and Kubernetes built in-house: maximum control and the highest engineering cost, because you own cluster upgrades, ingress, autoscaling, observability, and on-call.
3. Managed Kubernetes (EKS, GKE, AKS, Scaleway Kapsule) run by your team: no vendor control plane above you, but you still build the developer-facing layer and the CI/CD glue.
4. An internal developer platform in your own account (Qovery, Porter, Northflank BYOC): git-push deploys while the cluster, databases, and secrets stay under your own cloud credentials, and the cloud bill and its discounts stay in your name.

The post-incident test is one question: if the platform vendor's control plane goes down, who still holds your secrets, and can you still ship a fix? On your own account, your credentials sit in your own secret manager, and kubectl plus your cloud console keep working even when the deploy tool does not.

Railway, Render, and Fly.io are good products with excellent developer experience, and plenty of teams should stay. My rule: one service and no compliance requirements, stay on a PaaS. Multiple services, real customer data, audit or data-residency requirements, or an existing cloud commitment, move into your own account.

How do I migrate production off Railway step by step without downtime?

Stand up the new environment in parallel with rotated secrets, replicate Postgres with logical replication, validate on a canary subdomain, then flip DNS with a 60-second TTL and keep Railway warm for 72 hours as rollback. That sequence needs no maintenance window for most applications.

1. Step 0, freeze and inventory: freeze scope for the cutover week and list every service, database, volume, cron job, domain, certificate, outbound integration, and every third party that allowlists your egress IPs.
2. Step 1, images: commit a Dockerfile per service and build images reproducibly in your own CI or the destination's builder.
3. Step 2, environments: create staging first, using rotated secrets pulled from your cloud secret manager, never copies of Railway's values.
4. Step 3, data: provision managed databases and start Postgres logical replication, or use pg_dump and pg_restore if you can accept a short write freeze. Treat Redis as a cache and let it warm, or import an RDB snapshot.
5. Step 4, private networking: grep every variable for railway.internal and replace each hostname with a Kubernetes service DNS name before you deploy.
6. Step 5, validation: test health checks, migrations, background workers, cron schedules, and outbound calls from the new egress IP.
7. Step 6, cutover: drop DNS TTL to 60 seconds at least 48 hours ahead, flip, then watch error rate, p95 latency, and database connections for the first hour.
8. Step 7, decommission: after 72 clean hours, delete the Railway services, revoke remaining tokens, remove the Railway GitHub App, and archive logs for your incident record.

Postgres logical replication has three limits you must plan for. The PostgreSQL docs state that "the database schema and DDL commands are not replicated", "sequence data is not replicated", and large objects are not replicated (PostgreSQL logical replication restrictions). Copy the schema with pg_dump --schema-only first, freeze migrations during the replication window, and reset every sequence on the destination at cutover:

SELECT setval(pg_get_serial_sequence('orders', 'id'), COALESCE(MAX(id), 1)) FROM orders;

Logical replication also needs wal_level = logical on the source and network access from the subscriber. Railway now recommends removing public database access, so either expose the TCP proxy only for the replication window with a dedicated, freshly rotated replication user, or fall back to pg_dump.

Egress IP allowlists are the single most common cutover blocker. Payment providers, partner APIs, and customer firewalls that allowlisted your old outbound IPs will reject traffic from the new cluster on cutover day. If you enabled Railway's Static Outbound IPs on the Pro plan, those are the addresses your partners allowlisted. Route egress through a NAT gateway with a static IP (AWS NAT Gateway with an Elastic IP, GCP Cloud NAT, Azure NAT Gateway, or a Scaleway Public Gateway) and get that IP allowlisted days in advance.

The rollback trigger: if error rate breaches your SLO burn threshold in the first 30 minutes, flip DNS back. To keep rollback a DNS change instead of a restore, start reverse logical replication from the new database to the old one at cutover, so Railway's Postgres stays current for the full 72-hour window.

What does running in my own cloud account cost compared to staying on Railway?

Moving into your own account swaps Railway's usage-based bill for raw cloud compute plus a platform fee, and for a small workload the totals land within about 25% of each other at list price. Owning the account wins on cost once you use the levers that only exist there: committed-use discounts, spot capacity, and auto-stop for non-production environments.

Railway charges about $20 per vCPU per month, $10 per GB of RAM per month, and $0.05 per GB of egress, with the $20 Pro plan fee credited against usage (Railway pricing). Railway bills what you actually consume, so lightly loaded services cost less than their allocation. That is a real advantage.

Illustrative monthly estimate at list prices as of October 30, 2026. Assumptions: 730 hours per month; 3 app services consuming 1 vCPU and 2 GB each, plus a non-production copy of the whole stack; Postgres at 1 vCPU and 2 GB; Redis at about 1 GB; 500 GB of internet egress. On your own cloud, that means one cluster with four 2 vCPU / 8 GB nodes (two for production, two for non-production, with non-production databases running as containers), a small managed Postgres, a small managed cache, one load balancer, and one NAT gateway. Storage, backups, and taxes are excluded everywhere.

Sources: Railway, AWS EC2, Amazon RDS, Amazon ElastiCache, Amazon VPC, Elastic Load Balancing, Amazon EKS, GCP Compute Engine, Cloud SQL, Memorystore, GCP network, Cloud Load Balancing, Cloud NAT, GKE, Azure retail prices API, AKS, Scaleway. Scaleway prices are in euros. Cells marked "Not verified" are ones I could not confirm on Scaleway's published pricing, so check them before you budget.

Three levers only exist when the account is yours:

• Committed-use discounts: AWS Savings Plans save up to 72% (AWS Savings Plans), GCP resource-based commitments up to 55%, or 70% for memory-optimized machines (GCP committed use discounts), and Azure Reserved VM Instances up to 72% (Azure Reserved VM Instances). The top rates need three-year terms.
• Spot capacity for non-production and batch: EC2 Spot runs up to 90% below on-demand (AWS EC2 Spot), and GCP Spot VMs up to 91% (GCP Spot VMs).
• Non-production auto-stop: a staging environment that runs 10 hours a day on weekdays is up 50 of 168 weekly hours, about 30%. Qovery deployment rules do this on a schedule, and the DIY version is a cron job plus a scale-to-zero controller.

Egress is the line where Railway looks better than people expect. At $0.05 per GB, Railway undercuts AWS at $0.09 per GB after 100 free GB, GCP Premium Tier at about $0.12 per GiB, and Azure at $0.087 per GB after 100 free GB at list price. If your workload is egress-heavy, model that line carefully.

The other side of the ledger is people. Someone has to run the cluster, and the median US salary for a DevOps engineer is $165,000 (Stack Overflow Developer Survey 2025), with Levels.fyi reporting a $171,200 median total compensation (Levels.fyi). Put that number next to a platform fee and do your own arithmetic.

Railway wins on cost for one small service with no managed database, low traffic, and no compliance requirements. Owning the account pays off for real production workloads: several environments, managed databases, and enough steady compute to justify a commitment.

How does Qovery fit after an incident like this, and what does it not do?

Qovery installs an internal developer platform into your own AWS, GCP, Azure, or Scaleway account, or connects to an existing Kubernetes cluster you already run, so your team keeps the git-push workflow it had on Railway while the cluster, the databases, and the bill stay under your own cloud identity. That answers the question every team hits after reading the comparison above: if we own the account, who runs the cluster?

What carries over from Railway without losing the workflow:

• Git-push deployments from GitHub, GitLab, or Bitbucket, per environment.
• Per-pull-request preview environments, the same habit Railway's PR environments built.
• Databases backed by managed cloud services (RDS, ElastiCache, Cloud SQL, Memorystore, Scaleway Managed Database), or container databases for non-production.
• Per-environment variables and secrets, and role-based access set separately for development, preview, staging, and production.

What Qovery takes off your plate: cluster provisioning, automatic Kubernetes upgrades on managed clusters, ingress and TLS, and scheduled auto-stop for non-production environments. Upgrades matter more than people expect. Kubernetes ships about three minor releases a year with roughly 14 months of patch support each (Kubernetes releases), EKS gives 14 months of standard support per version (Amazon EKS docs), and AKS follows a 12-month policy (AKS docs). That is a recurring obligation, and it is the main reason teams choose Kubernetes without managing it directly and production without a dedicated DevOps hire.

On secrets, here is exactly where they live. With the Secret Manager integration, available today on AWS and GCP managed clusters, External Secrets Operator pulls values from AWS Secrets Manager, AWS Parameter Store, or GCP Secret Manager straight into your cluster, and the raw values never pass through Qovery. Qovery's built-in secrets do pass through our API and storage layer, so after an incident like this one, I would put production credentials in your own secret manager.

What Qovery does not do: it is not an all-in hosting bill. You pay your cloud provider directly, and you still own the architecture decisions about regions, database sizing, and networking. Porter and Northflank also offer BYOC, and raw Terraform plus Argo CD on managed Kubernetes is a legitimate path if you have platform engineers to spare.

How do I run this migration from Claude Code or Cursor?

Install the Qovery Agent Skills once and your coding agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI) handles the mechanical parts of the migration: writing Dockerfiles, creating environments, wiring rotated variables, deploying, and opening preview environments, proposing each action for you to approve before it runs.

Install globally for every compatible agent:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope it to the current project:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the MCP Server to Claude Code (OAuth, read-only by default):

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want it (this also requires enabling write access in the Qovery Console):

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) set things up, and the MCP Server operates what already exists. Then ask in plain language, for example "containerize these five services, deploy them to my own AWS account with Qovery, and create a preview environment per pull request". The agent proposes each step and you approve it.

Full setup: Qovery AI agent quickstart and Claude Code integration.