Solutions
Northflank Breach: Assume Your Secrets Are Decrypted and Rotate Everything
If you run anything on Northflank, rotate every secret now. The developer cloud platform told all of its customers on October 30 to assume their environment variables have been decrypted and to disable or rotate all keys, after users reported that their AI API keys were being used without permission.
Northflank runs customers’ containers, databases, jobs and builds, and environment variables are where applications keep their most sensitive material: API keys, database credentials, access tokens and cloud secrets. That is why the company’s guidance is not limited to one key or one service. It covers everything stored on the platform.
What happened, hour by hour
Northflank’s own status page gives the timeline in UTC on October 30. A threat alert from the security firm Wiz arrived at 11:43, engineers were reviewing it a minute later, and the first network policy patch went out at 12:02, with further patches at 12:50 and 13:05. Northflank asked Wiz’s incident response team for help at 13:00. At 13:40 the status page told customers to rotate environment variables immediately, and at 13:58 it went further: assume all environment variables have been decrypted, and disable or rotate all keys.
At 15:12 Northflank enabled secret rotation for PostgreSQL and MongoDB add-ons for all users. A 19:28 update said the investigation had no further confirmed findings and the guidance was unchanged. At 23:38 the company reported unauthorized use of AI API keys supplied to the platform, saying the reports may be consistent with the broader pattern of attacks targeting AI credentials in Wiz’s “Black October” report. At 01:04 on October 31 it switched on audit logs for every customer, including free accounts. The status page still showed major outages across networking, certificates, services, add-ons, jobs, builds and logs on the morning of October 31.
The rotation checklist
First, revoke each credential with the provider that issued it, not just inside Northflank. Changing a key in the platform while the old key still works at the provider leaves the stolen copy usable. Generate a replacement, update the secret, and update every workload that uses it.
Second, cover all three places secrets live on Northflank: secrets set on workloads, secret groups, and global secrets. Rotate database credentials with the new PostgreSQL and MongoDB rotation, and disable public access to add-ons where that option is enabled.
Third, check your AI provider usage and billing logs for activity since October 30, and report anything unexpected to Northflank and to the provider. Start with keys that bill per use, such as AI provider keys, because stolen ones get tested fast. And do not stop at AI keys: database URLs, payment keys, cloud credentials, webhook signing secrets and wallet keys all count.
What is not known yet
Northflank has not said how the attacker got in, how many customers were affected, whether data inside databases was read, or whether every environment variable was actually copied. No post-mortem has been published. The “assume decrypted” guidance is precautionary, not confirmation that every customer was compromised — but the reports of AI keys being used show real consequences for some customers already.
Even if you are not a Northflank customer
Wiz’s October 30 “Black October” report lists at least nine AI developer platforms that disclosed security incidents in October, with attackers going after stored environment variables, testing stolen keys within about 30 minutes, and later reselling them. The rule that follows is simple and applies everywhere: if any platform that stores your environment variables discloses an incident, rotate first and investigate later.