If you own a Google Cloud billing account where anyone holds an API key to an AI service, this text is for you. Google recently added spend caps to Cloud Billing, a mechanism that automatically pauses a covered service once it passes a set amount. Combined with the cost anomaly detector FOTC built (Cloud Anomaly Detector), it gives you full visibility into the account’s finances – the cap guards the set amount, and the detector shows whether the spend is normal and what caused the change.
How Cloud Anomaly Detector helped stop an LLMjacking incident within hours, we described in the previous article of this series. Today we go deeper – you will learn what a spend cap actually gives you, where its limits are, and how to roll caps out across dozens of projects without risking a production stop. You will also get three commands to check your own account today.
Since the first article was published, the system has run a full month on hundreds of billing accounts, was renamed Cloud Anomaly Detector and gained a new view in its dashboard dedicated to spend caps: a map showing which projects already have a cap and which still stand without one. We start with the caps, because they change the balance of power.
Spend caps: how the mechanism works
Budgets in Cloud Billing have existed for years, but they only notified. A spend cap is a budget with enforcement: once the estimated cost of a given service in a given project crosses the amount, Google stops new requests to that service. Nobody has to click anything. If you have ever seen the invoice for three days of runaway inference, this is a qualitative change. One cap covers one service in one project, and the list of supported services is short for now: Gemini API, Gemini Enterprise Agent Platform (formerly Vertex AI), Cloud Run and Cloud Run functions. A cap runs on a monthly period only, and since 13 October 2026 the whole feature is generally available (GA). That is the documentation as of 23 October 2026. Google says later releases will bring more services under caps, and each of them is one more area you can stop this way.
Cloud Anomaly Detector is a system we built that compares every billing account’s spend with its own history every half hour and raises an alarm when something deviates from it. It stops nothing by itself. Instead it answers the question a cap never asks: is this spend normal, and where did it come from?
What the cap blocks
- New API calls to the specified service in that project
- Until the start of the next month or until someone edits the budget by hand
What keeps running
- Compute, storage and other persistent resources
- Other projects and other services
- The cost of in-flight requests; enforcement is not instantaneous
We tell everyone with at least one AI service API key in their organisation the same thing: wherever Spend Caps are available, put a cap on every project where the service is enabled, and on every project where it is not. The second case matters more. The May incident we described in the first article was an attacker enabling an AI service in a project that had never used it. A hundred-euro cap on a project that has no business spending a cent on AI helps limit the cost of that kind of abuse. The amount has to be chosen deliberately, allowing for the delay in enforcement.
A cap on the project and monitoring across the account
A spend cap protects one project × service pair. Monitoring the whole account also covers the cost of newly linked projects. Google Cloud’s default configuration grants the Project Creator role to every user in the domain, and Billing Account User tends to be handed out more widely than anyone remembers. With permissions like that a script can create a new project, link billing and start an AI service. All your caps and project budgets are sitting in the old projects.
In one of the intrusions we were called in to help with, the attacker created a new project and linked it to the victim’s billing account. The client saw a bill they could not explain and asked us for help. The first question was not “who”, it was “which project”. Until a cost has an address, there is nothing to look for in the logs.
Once the address was known, the audit logs filled in the rest. The environment where the cost was growing belonged to nothing the client recognised, because somebody had created it from scratch and linked it to their billing account. That is why it pays to watch changes across the whole account, not only in the projects you know. A new cost line that spends from its first hour is a separate anomaly type for the detector.
Hence the rule we adopted for the whole portfolio: a spend cap enforces a set amount, while detection at the billing account level recognises unusual changes, including in new projects. Add two cheap IAM changes: take the default Project Creator role away from the domain, and review who really holds Billing Account User.
When caps and a manual review are enough
Not every environment needs a detector. If you have a handful of projects, one team and Project Creator revoked from the domain, then caps on the supported project × AI-service pairs, a budget on the whole account and a regular spend review in the console are enough for day-to-day cost control. The project list fits on one screen, and somebody will recognise every new line on the bill by name. The advice stops working once there are dozens of projects and the list of people who can create them is longer than the list of people who know it. At that point nobody holds the whole account in their head, and you need a system that remembers it for them. That is what the rest of this text is about.
How we approached caps at scale
With hundreds of accounts and thousands of projects you need a map. We built it as a second tab of the detector’s dashboard. A few decisions we made then do not depend on scale and work the same with a handful of projects as with thousands.
First, we focused on the denominator, meaning what enters the coverage calculation at all. We count every project linked to the account, including the ones that have never touched AI. If we counted only the projects already using AI, the map would glow green and say nothing about risk. You are protecting what the attacker is about to switch on.
Second, we decided that a budget without a service filter is not a cap, it is an alert, and it must not count towards coverage. When we counted it naively, coverage came out dozens of times better than it really was.
Third, we settled where the amount comes from. The account’s own history proposes it: the larger of two values, the average of full months and the last full month. The current month is excluded, because a limit computed halfway through the month is computed from half the traffic. We always round up, since a limit rounded down sits below the traffic it was sized for. A project with no spend gets a symbolic amount.
Fourth, we settled the order. We work the list from the smallest environments, because that is where a mistake costs least and where most of the history-free projects are. Large environments with real AI traffic get a cap only after a conversation about the amount. A badly chosen limit stops a client’s production as effectively as an attack does.
And the rule we do not break: what the API confirmed and what a person ticked by hand in the console have separate colours on the map and are never summed into one percentage. A declaration is valuable because it carries a name and a date. Only a readout is proof.
A month in production: between “deviates” and “worth waking someone up”
The detector compares every account only with its own history, so in every run a dozen or more accounts deviate from their norm, and that is how it should be. The whole value of the system is in how it picks, out of those observations, the dozen or so per month that justify interrupting somebody’s day.
A concrete case from 3 September. An account that normally spends a dozen or so euros a day spent 500 in a single day. A multiplier of ×23, growth of 2200 percent, practically all of it on one mapping service. The detector saw it at 03:00 and confirmed it 42 times over the next 21 hours. It woke nobody, because 500 euros does not cross the amount threshold at which we are allowed to interrupt someone’s day.
The alert did go out the next morning, from a different detector. The end-of-month forecast reached 6000 euros against 900 in the preceding months. At the scale of a day the amount was too small to wake anyone. At the scale of a month it was not.
It was not an attack. Part of the client’s mapping setup had failed, traffic had rerouted to a spare project, and it started generating cost where nobody was watching. A failure, not a break-in, but the bill would have grown just as long as nobody looked.
An alert has to point a finger
Two of the sixteen alerts ended in a joint session with the client. Both looked the same: the card said the account was spending many times its usual, and on the other side nobody could say which project the cost was coming from or which key was behind it. A billing account is too coarse an address to act on.
So detection moved further down, to a single project and a single SKU, the smallest line Google prints on an invoice. Services on the card are ranked by the amount they actually grew by, not by percentages. A service that jumped from one cent to seven euros grew by seven hundred percent and means nothing. A service that grew by seven thousand euros is the whole event.
The breakdown is computed in the same run as the rest of the card, and that is not an implementation detail. The billing export keeps filling in the window for hours after the alert, so services computed when the card is opened would come from a later state of the data than the amount in the headline. The card has to be a snapshot of one moment, even if that moment is incomplete.
A baseline that survives what it measures
First without the maths. Every day the detector answers one question: is today’s cost normal for this account? It needs two numbers for that: what normal is, and how far a day usually strays from it. The trouble starts when the past week already contains one bad day.
An example. An account spends about 20 euros a day. On Monday somebody runs inference worth 900, on Tuesday 640. Any method catches Monday, because 900 against 20 is a different world. Tuesday is the interesting one.
With the mean. The mean of the last seven days, Monday’s 900 included, is 146. The usual spread, computed from the same days, grew from 2 to over 300, because Monday stretched it upwards. Tuesday’s 640 sits less than two such spreads away from the new normal. To the statistics it is an ordinary day. The alarm goes quiet in the middle of the incident.
With the median. The median is the middle value after sorting. The seven days are 17, 19, 20, 21, 22, 23 and 900; the middle one is 21. Monday stands at the end of the list and has no say. The spread uses the same trick: take each day’s distance from 21 and again the middle one of those, which is 2. That is MAD, the median deviation. Tuesday’s 640 is several hundred such spreads away from normal. The alarm stays on, as it should.
| Mean | Median | |
|---|---|---|
| Normal after Monday | 146 | 21 |
| Usual spread | over 300 | 2 |
| Tuesday at 640 | almost an ordinary day | alarm |
The spread has a minimum, because an account spending exactly 20, 20, 20 would have a spread of zero and every cent would look like a catastrophe. And a day with no row in the export counts as a cost of zero, so a quiet week does not vanish from the history. The sliders under the chart show both lines on the same series.
Two adjacent traps. The current day: at six in the morning an account has six hours behind it, so against full days it always looks like a drop. So we compare today’s six hours with the same six hours of the previous days. Hourly detection: if the hour being scored sits in the same sample its normal is computed from, it raises that normal itself and never strays far enough to cross the threshold. An hour worth half a million then looks mildly elevated. The scored hour has to be left out of the sample.
The system has to know what it cannot see
The most dangerous state of an anomaly detector is silence that looks like calm.
On 25 August the billing export stopped writing. Not one source, all of them, one after another within tens of minutes. The detector kept running every half hour, scanning tables that had not changed since morning, and reported no anomalies every time. For six hours nobody was watching the fleet, and the only trace of that state was a strip on the dashboard that somebody had to look at on their own.
We fixed it the same day and immediately overdid it: a dead source started waking the on-call engineer. That night, just after midnight, on-call took a call about a frozen export table they could do nothing about until morning. The next day we removed that path. The pager is now reserved for cost, while a source failure has its own channel and a threshold counted in hours of silence, because a source flickering for a quarter of an hour is not an incident and one silent for half a day is.
Four states that naive code reports identically as “no anomalies”:
The export has not written for hours
The detector scans frozen tables and reports success. What is needed is a separate metric: how long ago the source wrote anything at all, with a threshold counted in hours of silence.
The query failed
An empty list because the query failed and an empty list because nothing was found are two different facts, and the code has to tell them apart. A run that did not see the fleet ends in an error, not a report.
An account without a full month of history
No forecast is produced, and rightly so, because there is nothing to produce it from. But the account is counted and visible as “not assessed”, with the date from which it will start being assessed.
Zero accounts instead of hundreds
An empty result means the run failed. An empty fleet only happens on a first deployment, and then it has to be declared explicitly.
Missing data is not zero. Every path that cannot prove it looked at data ends the run with an error visible in monitoring. The alternative is a green dashboard over a fleet nobody is watching.
How to check this yourself
Three commands answer the question this text is about: could a project nobody knows of appear under your billing account? The first lists the projects linked to the account with their creation dates, newest first. The second shows who in the organisation can create projects. The third, who can link them to billing.
$ gcloud billing projects list --billing-account=0A1B2C-3D4E5F-6A7B8C --format="value(projectId)" \ | xargs -I{} gcloud projects describe {} --format="csv[no-heading](projectId,createTime.date('%Y-%m-%d'))" \ | sort -t, -k2 -rgen-ai-tmp-4471,2026-09-10 ← created yesterday, zero cost historyshop-prod,2023-04-12shop-staging,2023-04-12data-warehouse,2022-11-03mobile-backend,2021-06-30 $ gcloud organizations get-iam-policy 123456789012 --flatten="bindings[].members" \ --filter="bindings.role:roles/resourcemanager.projectCreator" --format="value(bindings.members)"domain:example.com ← the whole domain can create projects (default)group:cloud-admins@example.com $ gcloud billing accounts get-iam-policy 0A1B2C-3D4E5F-6A7B8C --flatten="bindings[].members" \ --filter="bindings.role:roles/billing.user" --format="value(bindings.members)"group:cloud-admins@example.comuser:dev-2019-contractor@example.com ← a key from years ago, nobody remembers whyserviceAccount:ci-legacy@shop-prod.iam.gserviceaccount.com
domain: entry under Project Creator means Google Cloud’s default: anyone in the company can create a project.The same three commands, ready to paste. Fill in the two variables at the top and paste the rest unchanged. The account ID comes from gcloud billing accounts list, the organisation number from gcloud organizations list.
export BILLING_ACCOUNT_ID="0X0X0X-0X0X0X-0X0X0X" # gcloud billing accounts list
export ORG_ID="123456789012" # gcloud organizations list
echo "== 1. Projects linked to account $BILLING_ACCOUNT_ID, newest first =="
gcloud billing projects list --billing-account="$BILLING_ACCOUNT_ID" --format="value(projectId)" \
| xargs -I{} gcloud projects describe {} --format="csv[no-heading](projectId,createTime.date('%Y-%m-%d'))" \
| sort -t, -k2 -r
echo; echo "== 2. Who can create projects in organisation $ORG_ID =="
gcloud organizations get-iam-policy "$ORG_ID" --flatten="bindings[].members" \
--filter="bindings.role:roles/resourcemanager.projectCreator" --format="value(bindings.members)"
echo; echo "== 3. Who can link projects to account $BILLING_ACCOUNT_ID =="
gcloud billing accounts get-iam-policy "$BILLING_ACCOUNT_ID" --flatten="bindings[].members" \
--filter="bindings.role:roles/billing.user" --format="value(bindings.members)"Review the project list
Every project you do not recognise gets an owner or gets deleted. Repeat weekly, ideally from a schedule with an e-mail.
Narrow Project Creator and Billing Account User
Revoke the default role from the domain, keep the admin group. The list of people who can link billing should fit on one hand.
A cap on every project × AI-service pair
Including where AI is not used. A symbolic amount there; where traffic is real, an amount derived from history and rounded up.
A budget on the whole account
With a threshold set from real spend, not a round number. It also covers the spend of new projects linked to this account.
These four steps take an afternoon and need nothing beyond the console. They close the scenario from this text in a small environment and make it much harder in a large one.
Summary
Spend Caps in Google Cloud automatically enforce a set amount for the services they cover. Set them wherever you can, with special attention to projects that do not use AI. Cloud Anomaly Detector adds a second layer of control: it compares the spend against the account’s history, shows what changed and wakes somebody who can react. An unusual cost can deserve attention even when it fits inside the monthly cap.
Spend Caps watch the agreed amount. Cloud Anomaly Detector helps you recognise an unusual cost and work out what to do with it.
The four steps listed above are enough to start. As the environment grows, check that the caps still cover what they should. The limit is where the question “is this an attack or a new feature” begins, with dozens of projects and keys handed out over years. At FOTC we do this as Cloud Anomaly Detector: billing account monitoring every half hour with alerts to your team and ours, and when an alert has already arrived and you are not sure what to do with it, also as hourly engineering support with no long contract. If you would rather do it yourself, everything above is enough to begin.
FAQ
Will a spend cap protect me from LLMjacking?
It will limit the bill in the projects you covered. It will not stop the attack: enforcement is not instantaneous, in-flight requests get invoiced, and an attacker who can create projects will create a new one. A cap, account-level detection and a review of the Project Creator and Billing Account User roles close three different gaps. None of them replaces the others.
Why Cloud Anomaly Detector if I have Spend Caps?
Spend Caps pause a named service once it passes a set amount. The detector looks for deviations from the account’s normal spend and helps point to the project and service behind the change. It can report an anomaly that fits inside the cap, provided it meets the alerting conditions. It reads the billing export, so detection time depends on when the data arrives. There is no guarantee an alert appears before Spend Caps act.
What amount should I set on a project that does not use AI?
A symbolic one, on the order of a hundred euros a month. Enough that nobody notices it during a legitimate test, and low enough to stop abuse before it grows. When the project genuinely starts using AI, you raise the limit deliberately.
Can a cap stop my production?
Yes, and that is its job. A badly chosen amount on a project with real traffic will stop legitimate inference, and the cap only lifts itself at the start of the next month. Before that a person has to lift it, and recovery takes up to an hour. That is why on production projects the amount comes from history and rounds up, and someone with access to the budget has to be reachable at the weekend too.
What if the billing export to BigQuery stops working?
That is the one state a cost anomaly detector cannot notice on its own, because missing data looks like no spend. Source freshness has to be a separate metric with a separate alarm, and the dashboard has to say “I cannot see” in a different colour than “nothing is happening”.
The description of spend caps comes from Google Cloud’s public documentation, checked on 23 October 2026. Since 13 October 2026 Spend Caps in Cloud Billing are generally available (GA), and the documentation announces a wider list of services. The numbers about the detector’s work are real and come from its logs. The coverage map and the baseline chart illustrate the mechanism and are not production screenshots. The article describes the system at version 6.x (2026).