In spring 2026, two companies fell victim to the same class of attack: stolen credentials, someone else’s API keys, and mass Gemini consumption on their bill. The first incident ended with a half-million-euro invoice. The second, two months later, hit an environment already covered by our monitoring and was stopped within hours. This is the story of what we built between those two dates, and why the hardest part wasn’t detecting anomalies, but deciding when an alert is truly warranted.
Half a million euro in three days
May 2026. A large technology group running e-commerce and streaming platforms, among other things, uses Google Cloud symbolically. Its core infrastructure lives elsewhere; GCP hosts only peripheral systems generating about 500 dollars a month.
In the middle of the month, the company co-hosts a large, heavily promoted tech event. The entire engineering team is on site with their hands full. It is precisely during those days that someone reaches for a leaked key of an old service account, then escalates their permissions to the Editor role in one of the projects, enables Vertex AI and starts running inference on Gemini 1.5 Pro. Not as a test. At industrial scale.
The result? The bill for GCP usage between May 14 and 16 comes to 500,960 euro. A thousand of this company’s monthly budgets, racked up in three days.
The most bitter part of this story is that the alerts formally worked. Google sent a notification about suspected abuse and above-average spending as early as May 14. But the first budget alert, against a 500-dollar average, quoted an amount that looked harmless in absolute terms. And the people who were supposed to read it happened to be standing on stage.
The full response started on May 17 and was exemplary: accounts disabled, malicious IAM bindings removed, the project frozen read-only for forensics, new security policies across the whole organization. The investigation confirmed no data had leaked and the damage was contained to a single project. But the meter had already stopped at half a million euro.
An ordinary day on this account costs a dozen-odd euro, so at this scale those bars are simply invisible. This is not a screenshot of a real console.
Chart data (table)
| Day | typical | May 14 | May 15 | May 16 |
|---|---|---|---|---|
| Cost | ≈ €16 | ≈ €292,000 | ≈ €168,000 | ≈ €41,000 |
Watching this case unfold let us define three key problems, which went straight into our backlog:
- Static budget alerts are blind in low-spend environments, because a percentage-wise gigantic increase initially looks like small amounts.
- Abuse notifications go exclusively to the environment’s owner. With standard settings, the billing partner sees nothing in this area until the client reaches out on their own.
- In this case, there was no automatic circuit breaker that would have choked that consumption off.
We decided that at our clients, incidents like this would be spotted first by us, not by the invoice. That is how Cloud Cost Sentinel came to be.
Cost is a security signal
The industry already has a name for it: LLMjacking. Stolen cloud credentials are used not to exfiltrate data, but for mass consumption of expensive language model APIs, often resold onward. The bill stays with the victim. Publicly documented cases run from a dozen to tens of thousands of dollars for individual developers and small teams. The companies described here were not exceptions, just the next entries in a well-oiled operation.
That is why at FOTC we treat billing telemetry the way SecOps treats logs: as an event stream under continuous analysis. A sudden cost increase almost always has a technical cause, and a good share of those causes are incidents:
Architecture: from the billing export to PagerDuty
We operate hundreds of Google Cloud billing accounts. Each one has a full billing export into a partitioned BigQuery table. That is, incidentally, the only thing the system requires from an environment. On that foundation runs the detector: a light, stateless Cloud Run service woken by a scheduler in short, regular cycles.
Data
Detection
Decision
Response
Every component is stateless. The system’s only state is the detection history written back to BigQuery. It powers the dashboard and the re-alerting decisions. Alerts also come to us, not only to the client.
Four questions we ask the data in every analysis cycle
Each anomaly type is a different question with a different reference window. The common denominator here is that every account is compared exclusively against its own history. The specific thresholds, window lengths and monetary floors are configuration parameters chosen per deployment, which is why we don’t spell them out here number by number.
| Detector | Question | Reference Point |
|---|---|---|
| Daily (z-score) | Does yesterday’s or today’s cost deviate from the norm? | A rolling window of recent days, excluding the current day; alarm on a statistically significant deviation above a monetary floor |
| Hourly spike | Is this hour drastically more expensive than usual? | The account’s hourly profile from recent days; alarm at a multiple of that hour’s typical cost |
| New account | Is a freshly created account spending big right away? | An account with no history that immediately generates high spend; one of the strongest abuse signals |
| Monthly forecast | Where is this month heading? | Month-to-date extrapolation against previous months; alarm when the forecast clearly runs away |
The heart of the daily detector is a single windowed query in BigQuery, and that part of the implementation is actually the easiest. The real complexity hides in the details: the current day deliberately stays out of the baseline (Why? You’ll see in a moment…), the windows and thresholds have to be tuned to real data, and the data itself must never be trusted blindly, neither its completeness nor its layout. These sound like trifles, but every one of them has hurt us at some point.
July: the same class of attack, a different ending
Two months after the May incident, the same attack scenario hit one of our clients, a small mobile app studio. That environment was by then already covered by Cloud Cost Sentinel monitoring. It began trivially: one of the developers wanted to install a popular tool and clicked a sponsored search result. The site looked right, so did the installer. Inside sat a stealer from a campaign publicly documented by Bitdefender and Trend Micro, among others. Even though the account had MFA enabled, it made no difference, because this type of malware operates from inside an already signed-in machine. It steals a session whose authentication is long behind it.
In the evening of July 4, a script logged in with the stolen credentials from a hosting server in Germany. After years of Firebase collaboration the account held the Editor role in a dozen-plus projects, which in Google Cloud means, among other things, the right to enable APIs and create keys. The automation needed 3 minutes and 16 seconds to walk the projects and leave behind 16 enabled APIs, 9 service accounts and 28 keys. Then silence fell for twelve days. The keys lay in wait.
On July 16 at 13:45 the keys came alive and mass Gemini consumption kicked off across all projects at once. At 4:30 in the morning, as soon as the first data for that day reached the billing export, the detector raised the alarm. The CRITICAL card looked like this: a z-score well past five hundred, growth measured in tens of thousands of percent, and in the per-service breakdown practically the entire increase came from a single line: Gemini API, a service never before used on this account. Escalation went out.
This client wasn’t on our standing support and hadn’t granted us permissions to their environment, so our role ended at the alarm and the escalation. Only they could block the abuse. The client’s team picked it up at the start of the day: around 08:20 traffic was back at the baseline, and by 12:40 the foreign keys and service accounts were gone from every project.
This account’s baseline is about 20 dollars a day. One day of active abuse was enough to exceed the account’s total yearly spend several times over. This is not a screenshot of the client’s console.
Chart data (table)
| Day | typical | July 16 | July 17 |
|---|---|---|---|
| Cost | ≈ $20 | ≈ $33,250 | ≈ $650 |
Once the export had filled in the day, the meter stopped at thirty-three thousand dollars, and the deviation grew to 842 sigmas. The day closed 906 times above the account’s norm. On the timeline it looks like this: consumption started after 13:45, the alarm went out at 4:30, because that’s how long the data took to reach the billing export, and everything else was reaction time on the client’s side. It was a Friday. The May story shows fairly precisely what would have happened had that consumption quietly worked through the weekend. Where we hold not just a mandate to escalate but also the permissions to react, those few hours can shrink to a few minutes.
The alert that kept putting itself out
On day two of the July incident the account was still burning 14.2 thousand dollars a day. And our z-score for that day came to… 1,1. Statistically? Nothing interesting, move along.
The reason: the rolling window had managed to absorb the peak day into its own baseline. The rolling average jumped from 20 dollars into the thousands, the standard deviation exploded, and the ongoing incident started to look like the new normal. A critical alert kept degrading itself into a warning even though spending hadn’t dropped a cent.
The first fix was statistical: exclude anomalous days from the baseline, computing it in two passes. It didn’t work. Day two of the incident already had a deflated z-score, so it escaped the outlier flag, and from day three it poisoned the window all over again. Only after the revert did we understand we were fighting the wrong enemy: this is not a statistics problem, it’s an alert lifecycle problem. An alert is a state, not a single event.
The episode lights up. A card in Chat, a task in the CRM, PagerDuty too for criticals. Written to the detection history.
Severity is a floor. A poisoned z-score can’t downgrade the alert or fire a duplicate. Silence, unless the incident genuinely escalates.
The episode dies down. After a few days with no new alert the tile disappears from the dashboard. A renewed escalation opens a new episode with a new clock.
Not every anomaly is an attack
Let’s remember, though, that the system doesn’t live on security incidents alone. One day, the daily cost of one of the accounts we monitor rose 7.6 times above the norm. The alert went out, a CRM task reached the account manager, and within a dozen-odd minutes he confirmed with the client that this was a planned model training run on Vertex AI. The episode was muted with a single click until the end of the day, with critical escalation kept intact. The client got a “Is this yours?” an hour after the fact, instead of a surprise on next month’s invoice. A system that can’t quickly tell the expected from the malicious teaches people to ignore alerts, and that is a straight road to a repeat of May.
Details that only start to hurt in production
Numbers without context lie
The same amount is noise in one environment and the incident of the month in another. A threshold compared against a raw number means, in practice, “1000 of whatever”. So we anchor every value in the context of the specific account, its scale and its history, and we show people amounts in the form they know from their invoices.
Data breaks silently
Billing data from different sources and periods can differ in layout and types. Code that assumes it’s the same everywhere won’t crash with any error, it will just quietly shuffle the numbers. In a system people are meant to trust, that is the worst kind of bug, so no assumption about the data is ever implicit here.
Alert fatigue is debt
Every needless alert erodes trust in all the following ones. That’s why the system carries a whole set of anti-spam safeguards: deduplication, cooldowns, muting and automatic episode expiry. We design for signal, not volume.
Billing arrives late
The export can keep “topping up” a day for many hours. The July card from 4:30 showed an amount several times lower than the day’s final close, and the meter kept climbing long after the traffic was blocked. The system has to know this, or it would mistake late-arriving data for escalation and wake people for no reason.
One engine, two modes of operation
The same architecture runs in two modes today.
The first is a managed service. We cover the client’s environment with our monitoring, alerts reach our team and the client in parallel, each anomaly is classified and owned by a specific account manager, and wherever we’ve been granted permissions, we react directly instead of merely escalating. The July story shows exactly where that boundary runs: without permissions, even the best detection ends with a phone call to the environment’s owner.
The second mode follows directly from the design. Since the engine is stateless and relies solely on the standard billing export to BigQuery, it can be stood up in any Google Cloud environment, including ones we don’t operate day to day. In such a deployment we narrow the alert scope from the whole billing account down to a project view. Each team sees the anomalies of its own projects, with its own thresholds and notification channels. Chat, e-mail or a webhook are interchangeable adapters from the system’s perspective. Everything else, from the SQL windows to the episode lifecycle, works identically in both modes.
What this means for your environment
Whether or not you build something like this yourself, a few things carry over directly. The baseline must be behavioral and computed per account, because a shared threshold for every environment is a fiction. An alert should have a lifecycle, because without the hold, deduplication and autoclose the system devours its own credibility within weeks. Statistics only detects, the decision is made by context, meaning the mapping of an anomaly to a client, an owner and specific services. And the simplest of the hard truths: cost is security telemetry. Finance looks at it once a month, monitoring has to run non-stop.