TL;DR Google is changing the pricing and compute model of BigQuery dramatically for both compute and storage. New orders of Flat-rate and Flex Slots will be discontinued. Migrations from flat-rate to this new billing model will happen in April and pricing changes will start on July 5, 2023.

BigQuery Editions was just announced by Google on March 29, 2023 and it’s a pretty massive change from the the pricing models that have existed for most, if not all, of its lifetime. In this article we will be covering the compute-side of BigQuery and what has changed with these announcements. My colleague Philipp has written an article covering the storage changes to BigQuery that accompany Editions here.

When looking at pricing it is best to look at pricing for both the compute and storage as the price increases to the compute outlined in this article will be offset, and in some cases eliminated, by the new storage changes.

Fair warning this is going to be a very long article and as your author I fully know that different pieces of it will be needed by different individuals in varying roles, so I recommend browsing through it to find what you need and go from there. I have put in all of our collective knowledge from my team here at DoiT International into this article you are about to jump into for all GCP customers (both our customers and the rest of that should come join our customer base!) after asking the BigQuery team every question we could think of that might come about on this.

Google has also posted their guide on choosing Editions and how to choose the sizing you need here.

Due to this being so long below is a Table of Contents for assisting in jumping around in this article.

Table of Contents

Key Terms to Know
Timeline
Editions? What is this madness?
Can I go Back From the Future?
What Other Changes are Coming?
What if I am on an Existing Commit?
End-of-an-Era on Flat-Rate and Flex Slots
What About On-Demand Usage?
Overview of Each BigQuery Edition
> Standard Edition (SE)
> Enterprise Edition (EE)
> Enterprise Plus Edition (EPE)
Which Edition do I Need to Use?
Autoscaling in BigQuery Editions
Reservation Structure
Commitments for Editions
Pricing Updates
Pricing SKUs
The Auto-Switch to BigQuery Editions
Mixing Flat-Rate and Editions with Autoscaling
In Conclusion

Key Terms to Know

Here is a quick primer on terms used throughout this article. If you are familiar with BigQuery then most of this will be known, if not then here is some assistance before continuing on.

Some old terminology:

• Job
  A job is an action that runs on BigQuery to perform work such as querying, loading data, copying data, etc. The most common type of job is a query and the two are used quite often in lieu of each other.
• Slot
  The basic compute construct used in BigQuery to do work for a job. Think of this as a “mini-virtual machine” that joins with many others to perform work for a job.
• Reservation
  A construct that reserves a set of slots to be utilized for zero or more projects that are added to it.
• Commitment
  A fixed number of slots that are reserved for a committed duration of time, usually one month or one year, in exchange for a cheaper price.
• Admin Project
  A project that is designated as the “manager” of BigQuery reservations and commitments. Generally one organization will have a single (or a select few) of these to allow a “single pane-of-glass” for managing the BigQuery resources of the organization. For flat-rate and Editions projects the reservation billing SKUs across the organization will be associated with this project.
• Flex Slots
  Slots that can be added to a reservation for a short term burst in slot counts. Note they are billed by the second with a minimum billing period of 60 seconds.
• Idle Slot
  A slot that is currently not being utilized in a job.
• Workloads
  A workload is a set of jobs that are run inside of a single project.
  The new terminology specific for Editions:

The new terminology specific for Editions:

• Slot/Hour
  The basic billing metric for BigQuery Editions projects. It is defined as a single slot being utilized in a job for one hour.
  Calculation for determining slot/hours used is:

Slot/Hour Used = <slots used> * (<duration in seconds>/3600 seconds)

• Autoscaling Slots
  A new construct released into public preview early in 2023 that scales the number of slots up from a baseline slots value to a maximum slots value as needed for a running workload.
• Baseline Slots
  The minimum amount of slots kept “hot” and active in a BigQuery Editions reservation. These will be utilized first before autoscaling occurs and are billed as long as the reservation is active.
• Max Slots
  The maximum amount of slots that will autoscaled to in a BigQuery Editions reservation. The number of slots allocated will never go above this number across all workloads inside of all of the projects inside the parent reservation.

Timeline
Google has released a timeline for when these changes are going into effect. This is of the utmost importance to know when things are going to change so you can be prepared.

March 29, 2023
Google announced BigQuery Editions and pricing changes. Note pricing will not go into effect till July 5th.

March 30, 2023
BigQuery Slot Autoscaling is considered Generally Available.

April 2023
BigQuery Editions will start rolling out to customers without any price changes.

July 5, 2023
End-of-sale for flat-rate and flex slots for BigQuery for all customers not currently in a contract.
Editions pricing will commence for all non-on-demand projects.
The new on-demand pricing of $6.25 USD/TB of data scanned will commence for on-demand projects.

Editions? What is this madness?

Google announced Slots Autoscaling earlier this year which was a subtle precursor to BigQuery Editions. BigQuery Editions can be defined as an overhaul of the BigQuery compute pricing model to shift from an always-on to a pay-per-usage model. This is called BigQuery Editions or Editions for short in this article.

As hinted at with Slots Autoscaling this is the foundation of Editions where going forward the standard billing model will be based upon the concept of a slot/hour, which as defined above is usage of a single slot for an hour time period.

Since it’s called Editions and not Edition Google has introduced three different levels of service, features, and pricing to choose from, conveniently and unsurprisingly called an Edition.

The most common question we were asked when Slots Autoscaling was released was: will this replace the current pricing models? It was an unknown at the time, but as part of this announcement Google has stated that Editions will replace the flat-rate pricing model, which also includes Flex Slots. This will be effective on July 5th.

Can I go Back from the Future?

Without a time-traveling Delorean, unfortunately if you are on flat-rate pricing currently Google will transition flat-rate pricing to Editions without a method to revert back to the previous pricing model.

This means that on July 5th, 2023 monthly commits will be automatically converted to the Editions pricing model, specifically Enterprise Edition mentioned below. Annual commits will continue on as this is a contract that prices will not change for a year.

Unfortunately if you are on an annual commit you are locked into that amount of slots and are unable to purchase flex slots or additional slots for your commitment after July 5th, 2023. If you need additional slots beyond that you will need to change to Editions, unless Google announces a middle ground in the near-term.

What Other Changes are Coming?

Before getting into an overview of each Edition I am wanting to answer some questions that are bound to be burning in everybody’s heads since by the time of this reading news outlets and Twitter will have spread the news far and wide. So let’s dig into these questions and answers for you so when your CEO calls you at 7 AM tomorrow morning wanting to know what this means you can answer those questions.

As stated above, the much-beloved by read-heavy users flat-rate pricing model is going to be going away to make room for Editions. In April, Google will be rolling every project on a flat-rate reservation onto an Edition called Enterprise Edition, or EE for short, which will be covered below. This will NOT affect pricing of those projects at this time.

This should not have any performance degradations to existing workloads and you should not see any changes from a workload perspective. We have been reassured by Google that this will be a seamless transition.
This will be running in such a manner to allow any modifications of the projects until July 5th when the pricing for Editions will officially start.

What if I am on an Existing Commit?

If you are using flat-rate pricing then you will have a commit, either monthly or annually. Monthly is the default so chances are if you are unsure you are on that or you can check your bill and look for SKUs named either “BigQuery Flat Rate Monthly for ” or “BigQuery Flat Rate Annual for ” which will let you know the length of the commit.

For customers on a monthly commit, which is the default, on July 5th these will be converted over to Enterprise Edition (more on this later in article) automatically.
If you are currently on an annual commit then the terms of that commit will not change until the anniversary of it in which all reservations attached to it will then change over to use Enterprise Edition.

End-of-an-Era on Flat-Rate and Flex Slots

On July 5, 2023 flat-rate pricing and flex slot purchases will be disabled for all users unless you have a contract with your Google accounts team explicitly allowing the use of them beyond this period. This also means that even if you currently are in an annual flat-rate commit you will be unable to purchase flex slots after July 5th as well.

I will take a moment to let all of us faithful users of flat-rate pricing and flex slots to collectively join together in a moment of silence for their demise.

Once that moment of silence is up let’s continue.
So this means if you wish to keep your flat-rate slots either to buy time to rearchitect your environment to take advantage of these changes or for other reasons it is best to purchase an annual commit prior to that date. It’s best to do this well before July 5th just in case there are changes to the announcement due to a “gold rush” on commitments.

What About On-Demand Usage?

An astute reader or casual BQ user will realize so far I have strictly brought up flat-rate billing changes and not on-demand pricing.

Currently BigQuery’s default pricing model is the on-demand model which charges $5 USD per terabyte of data scanned by queries or jobs, also called analysis in the billing and by many users. Starting on July 5th this will be raised to $6.25 USD per terabyte of data scanned in the US and certain other regions will be slightly higher.

The existing features and limits will not change at all for customers choosing to stay with on-demand. I will cover this later, but on-demand pricing will have the same feature set as the Enterprise Plus Edition but is limited with the traditional limits such as the 2,000 query slots. It has been asked a few times, but BigQuery ML (BQML) is available to on-demand billing still.

A question that has come up a number of times already about on-demand and Compressed Storage: when using Compressed Storage and on-demand are you billed for the logical (uncompressed) or physical (compressed) storage scanned? The answer is that it bills on the uncompressed storage as BigQuery must uncompress it before analyzing. This value is reflected in the bytes_billed values in INFORMATION_SCHEMA and the audit log sinks.

Edit: Previously I had stated that on-demand will not work with Compressed Storage, but that was incorrect and after doing some research I have discovered this was incorrect. If you are “Edition-eligible” meaning no flat-rate projects in the reservation, have an Edition reservation, or are on an on-demand billing model Compressed Storage is available.

Overview of Each BigQuery Edition

One of the strongest points Google has been pushing on these Editions is that this affects only the compute pricing side of BigQuery not the storage half. Storage is not affected by these changes at all. So keep that in mind as you read through the rest of this.

Before looking too deeply into these, note that a single Edition can be applied to a single reservation which may contain zero or more projects. So this means you can mix-and-match Editions as you see fit across reservations and projects for optimal usage or costs.

Standard Edition (SE)

• Autoscaling of slots
• Capped at 1600 slots per reservation as a hard limit
• Max of 100 concurrent queries
• 5 max reservations per admin project
• Runs compute workloads in a single zone only
• HIPAA Compliant
• Google managed keys
• Only allows a maximum of 5 reservations per project with the type of SE
• Cannot run BigQuery ML workloads
• SLA is only 99.9% so may not be advisable for production workloads

Enterprise Edition (EE)

• All features of Standard Edition
• No slot or concurrent queries limit
• Guaranteed minimum/baseline and reserved capacity
• Query acceleration
• Cross-user result cache
• Allows running of BigQuery ML workloads
• Support for object tables
• VPC Security Controls to prevent data exfiltration
• Support for data masking as well as column and row security
• Zonal compute redundancy and disaster recovery (if one zone is down can run jobs on another zone in the same region)
• Allows jobs to be run across cloud provides (BigQuery Omni)
• 200 max reservations per admin project
• SLA is 99.99%

Enterprise Plus Edition (EPE)

• All features of Enterprise Edition
• Customer-managed encryption keys
• Support for FedRAMP, ITAR, and other compliance requirements available through Assured Workloads
• Regional compute redundancy and disaster recover (if one region is down can run jobs on another region in the same multi-region)

Which Edition do I Need to Use?

As happens WAY too often in the world of cloud, the answer is it depends. That answer sucks, so let’s dive in and ask a few basic questions to get you started on the right Edition.

The first question I will pose is do you need the compute of your workload to be run in more than a single zone? If the answer is yes then you can immediately cross out the Standard Edition for your workload as this is constrained to a single zone. Note again this is for the compute aspect of your workload, not your storage.

The next question is will your workload need more than 1,600 slots? If the answer is yes, it crosses out the Standard Edition as well. Google has shared that the 1,600 slots is a hard limit and will not allow burst flexibility like their current on-demand 2,000 query slots does.

The next question is do you plan on using BigQuery ML in any of your workloads? If so then Enterprise Edition is the minimum version that supports this.

Some other things to consider is if you are constantly hitting the 100 concurrent query issue very often then I would recommend looking at Enterprise Edition as they have removed that limit there.

An increasingly more common scenario is if you need to manage your own encryption key then it will require the Enterprise Plus Edition for that.
The biggest thing to keep in mind is that each project can be on-demand or have a different Edition applied to it. This will help out each user optimize their usage and costs on a per-project, or per-reservation, basis.

For DoiT customers feel free to open up a ticket with a member of the Customer Reliability Team for a call or analysis of your usage.

Autoscaling in BigQuery Editions

One of the big selling points of Editions is slot autoscaling which means it can scale up from your baseline value to the max slot value when slots are needed.

This is immensely powerful if you currently use flat-rate billing and need more slots than you have reserved. Flex Slots gave you an approach to getting more slots, but they were not automatic and required additional tooling, usually in the form of a Cloud Function running on an interval that checked the used slots metric and added or removed Flex Slots to a reservation as needed. Unfortunately if more slots were needed immediately you were out of luck unless you built and paid for additional infrastructure to do this.

Enter Editions where this is now done automatically for you and is handled by GCP for scaling up and down as needed on a per-job basis.

Now with that said there is a 60-second window for scaling where it does the determination of how it will scale. So this prevents short-run queries to prevent it from scaling up to the max when slots won’t be used for less than the minimum billed amount. This means if you have a query that runs for 10 seconds it probably won’t scale up to your max slots, or maybe not even your baseline depending upon complexity.

Note that both the baseline and max slots represent a shared pool of slots across all jobs running in projects inside of a reservation, which is the same as a flat-rate reservation. When the slots are no longer being used by a job they are scaled back down so you aren’t billed for them which is the differentiator from Flex Slots that are billed till they are manually removed.

One final note on autoscaling, slots are not guaranteed to be available unless they are inside of a commitment, covered later, and are subject to availability of the region or multi-region they are located in.

Reservation Structure

Reservations under BigQuery Editions are not that different from previous reservations for flat-rate billing, but they do have some new concepts associated with them which need to be covered to fully understand them.

They still are created inside of a management project and inside of each reservation are zero or more projects to which slots are allocated for use. Note above in the Editions comparison, there are now limits on the number of projects a reservation can contain based upon the type of Edition.

Since Editions introduces a concept of autoscaling there is now a minimum scaling value, called baseline slots, and a maximum scaling value, called the maximum reservation size. Note that baseline slots do not exist on Standard Edition, presumably due it having a slot cap on it, and this value is always zero.

The maximum value is defined as the baseline slots value plus the number of autoscaled slots inside of the value for the entire reservation.

By default if a job, or set of jobs, overflows the max autoscaling amount for its reservation then it can borrow slots from other reservations from the same management project using the same Edition, just like a standard reservation. There is a checkbox that allows this feature to be turned off in the “Create Reservation” screen:

Commitments for Editions

In keeping with tradition BigQuery Editions carries forward the concept of a commitment for a term in exchange for a lower price. These commitments are created for a fixed amount of dedicated slots that are paid for 24/7, but at a lower rate. In regards to billing they are just like having a Baseline Slots value set which you pay for each month up until the term is over, but just at the lower billed rate.

Note that the benefit of committed slots on Editions is to get a cheaper set of slots dedicated to your reservation that you will be using 24/7. Slots that are not used all-of-the-time should not be used in a commitment as you will be paying for resources not being used. Slots that are committed are guaranteed to be available.

Unlike on flat-rate where you were able to choose a monthly or annual commitment, Editions commitments are annual only.

As previously on flat-rate commitments were on multiples of 100 slots, the new commitments are likewise on multiples of 100 slot/hours.
The gotcha on them is that they are only applicable to Enterprise and Enterprise Plus editions. Pricing is in the next section.

Pricing Updates

Pricing is almost completely different from the traditional pricing models BigQuery has had for a while now. The new pricing model is based upon a concept called a slot hour (or as we are calling it which is more descriptive a slot/hour), which is a charge per slot for an hour of usage billed by the second.

Editions bills your usage based upon how long you use a slot for, note this is billed by the second but uses the hour as a nice rounding point on the numbers. So if you use a single slot for 900 seconds (1/4th of an hour in seconds) you will be billed for 0.25 slot/hours or using it for 3,600 seconds (1 hour in seconds) you will be billed for 1.0 slot/hours. Hopefully you will be using more than a single slot for a job, so multiplying this by the number of slots used will tell you how many slot/hours you are billed for.

The missing piece to determine price here is the cost per slot/hour which differs per Edition chosen. All prices are in USD for the US multi-region and may be different in other regions.

At launch the pricing is as follows for each Edition:

• Basic Edition: $ 0.04 USD per slot/hour
• Enterprise Edition: $0.06 USD per slot/hour
• Enterprise Plus Edition: $0.10 USD per slot/hour

So the overall formula for determining the cost for a job is:

Price = <Per slot/hour fee> * <Slots used> * (<Duration in seconds>/3600)

Essentially you are multiplying the slots used by the hours it was used (seconds used / 3600 seconds) then lastly by the Editions charge. It looks more difficult than it is.

One final note on this is that there is a 60-second minimum billing period here. This acts the same as flex slots where you can use it for less than 60 seconds, but you will be billed for 60 seconds.

Commitment pricing is as follows, keeping in mind that 100 slots is the minimum size and assuming 730 hours per month:

Enterprise Edition

• 1 year:
  $0.048 USD per slot/hour
  $4.80 USD per 100 slot/hours (hourly)
  $3,504 USD per 100 slot/hours (monthly)
  $42,048 USD per 100 slot/hours (annually)

• 3 year:
  $0.036 USD per slot/hour
  $3.60 USD per 100 slot/hours (hourly)
  $2,628 USD per 100 slot/hours (monthly)
  $31,536 USD per 100 slot/hours (annually)

Enterprise Plus Edition

• 1 year:
  $0.08 USD per slot/hour
  $8 USD per 100 slot/hours (hourly)
  $5,840 USD per 100 slot/hours (monthly)
  $70,080 USD per 100 slot/hours (annually)

• 3 year:
  $0.06 USD per slot/hour
  $6 USD per 100 slot/hours (hourly)
  $4,380 USD per 100 slot/hours (monthly)
  $52,560 USD per 100 slot/hours (annually)

Pricing SKUs

Slot usage under Editions is billed to the management project itself, not to projects that start the jobs as it would be in the on-demand billing model. This is the same as it is under flat-rate billing thus making reporting changes minimal to reflect the new SKUs migrating from flat-rate. These SKUs are under the BigQuery Reservations API grouping in the billing data.

SKUs are named like this with different edition and region names as set:

• BigQuery Enterprise Edition for US (multi-region)
• BigQuery Basic Edition for US (multi-region)

For commitments the SKUs are named like this, again with different edition and region names:

• BigQuery Enterprise Edition 1 Year for US (multi-region)

The Auto-Switch to BigQuery Editions

On July 5, 2023 every non-contracted and non-annual commit flat-rate project will be converted over and started to bill under Enterprise Edition’s billing scheme. Google has assured us this will be a painless process and it will not impact any workloads.

To switch from on-demand to Editions (or flat-rate) is a bit more of a process. I would highly recommend reading Google’s documentation on the subject here before switching and then following the procedures to create a reservation, but selecting your chosen Edition instead of flat-rate in the menu.

Mixing Flat-Rate and Editions with Autoscaling

This is a new section I wanted to add as this has been in place since late April, but the official documentation hasn’t been updated to explicitly call this out and some customers . This was explained to us during a call with the BigQuery product team after doing some experimentation and only seeing a reference to it on the “Create Reservation” page (see the “exclamation” note in the screenshot below).

You are able to have a flat-rate reservation and add autoscaling Enterprise Editions slots on top of it.

Wait what!?!?! Yes, this is possible since in April Google migrated all flat-rate reservations and commitments over to being run on the Enterprise Edition backend, so they are essentially Enterprise Edition reservations and commitments just with flat-rate pricing. This means you can pay for flat-rate but have some autoscaling to make up for the loss of flex slots until July 5th or your annual flat-rate commitment is up.

This works inside of a reservation by treating your flat-rate commitment slots as a baseline slots level in Editions, which are billed at the flat-rate price, and scaling up to the maximum reservation size, billed at the EE price. Since on the backend it’s all running on Editions this works just like with a vanilla Enterprise Edition reservation.

Let’s dig into the details of how to do this with an example.

Currently, there is a project running that uses 1,000 slots inside of a flat-rate reservation and has a monthly ETL job that runs that has been seen to scale up to 1,500 slots for the job’s duration.

What you would need to do is edit the reservation, or delete the old/create a new reservation, setting the baseline to 1,000 slots and the max reservation size to 1,500 slots such as shown below. Not if you aren’t seeing these values, select the “Custom” reservation size in the dropdown.

Saving this will then create a reservation with 1,000 slots that are the baseline, but billed at the flat-rate price, and an additional 500 slots that can be autoscaled to then billed at the Enterprise Edition rate when utilized.

This will act as a good compromise for customers wanting a “life jacket” with the annual flat-rate commitments beyond July 5th and being able to still scale when needed. While the scaling slots are more expensive than Flex Slots, they have taken away the need to manually create and delete the reservations for them.

Note that this behavior will work with all monthly flat-rate commitments until July 5th when the ability to create new monthly commitments is removed.

In Conclusion

All of us at DoiT International do realize this is a massive paradigm shift in the pricing for BigQuery so we want to share this knowledge above along some final warnings.

Make sure you are 100% prepared for this change on July 5th, 2023. Especially if you are on flat-rate billing as switching from flat-rate to Enterprise Edition may cost you significantly more money. If you are using on-demand just know it’s a flat 25% increase so reducing scan costs in advance will help offset this.

We recommend you find every project with BigQuery billing on it inside of of your organization and discover if it is on flat-rate or on-demand billing models. This will help prevent any unknown jumps in cost that you weren’t aware of. This is best done through the Billing Explorer or the DoiT Console for DoiT customers.

Lastly I HIGHLY recommend doing the cost calculations according to your slot usage and then look at your storage costs to see if using compression will assist there. With such a price change without some planning there can be massive jumps in pricing, so you need to be prepared for this. While there isn’t much time from the writing of this article till July 5th, but in that time being prepared even a little will pay huge dividends especially in having a long-term design in mind.

As a very final note I have to do the obligatory plug about my company DoiT International. Assisting customers doing analysis and long-term planning after changes like this is pretty much one of the major things we do here. So if you aren’t a customer yet then give us a look as we are awesome (well yes I know I am biased here but I honestly do think so) and can help you and your company out on this sort of thing.