TLDR; BigQuery now offers a new storage pricing, which is based on physical storage, after compression. With compression ratios up to 30% this new pricing can reduce your Storage bill significantly.
What is BigQuery Compressed Storage?
A few months back, Google announced BigQuery “Physical Storage Pricing” in a private preview. Today, at Google Cloud Data Summit, this feature was announced to go into General Availability on July 5th, 2023!
It is now available for every customer using one of the three Editions, or any exclusive on-demand customer, under the new name “Compressed Storage”.
This feature can reduce your BigQuery bill drastically if you store lots of data in BigQuery.
During the Preview, some customers achieved up to 50% savings while retaining full performance. For more details, see our latest case study with Plegendary.
In January, you may have already spotted a new SKU description for Storage on your GCP bill or in the DoiT Console (if you work with us):
This was just a name change; “Active Storage” has been renamed to “Active Logical Storage” and “Long Term Storage” has been renamed to “Long Term Logical Storage”.
After you enable compressed storage billing model, the storage price will be based on the compressed bytes used by the tables in that dataset. Data compression is handled automatically, and you can’t fine tune or adjust it. In our internal tests, we have seen compression rates in the 1:6–1:12 ranges. However, this varies a lot depending on data volume and data modelling. There are no performance penalties, and queries will be just as performant as before: it’s only the storage pricing model that changes.
The compressed storage billing model has higher– roughly double– the storage costs per original gigabyte, compared to the standard logical storage billing model. It still saves a lot of money, as the storage reduction itself is much larger than double.
Some changes have been made to the BigQuery UI, to inform the user about the new billing model:
- Each Table now has detailed information about the Compressed Storage Size, Logical Storage Size and Time Travel Storage Size
- The Information Schema views have been updated, and now have a dedicated storage-view.
What does BigQuery compressed storage cost?
Compressed Storage will be listed under new SKUs. For US multi-region, you pay $0.04 per GB for active compressed storage compared to $0.02 per GB for Long Term storage. Please refer to the official pricing page to determinate the actual pricing for your specific region.
(Note: the SKUs will eventually be updated to reflect the latest rebranding to compressed storage)
To get a better grasp, how much you could save, we have prepared a query which lists the costs per dataset.
Keep in mind, that (compared to the Preview) a 7-day fail-safe period has been introduced, which will retained deleted data for an additional 7 days in fail-safe storage after the time travel window. It is non-configurable you have to reach out to Google to request deleted data, it cannot be queried.
Am I eligible?
Customers who exclusively use the on-demand model, and who have completed migration of their flat-rate slot commitments to BigQuery editions are eligible to use the new physical storage billing model, where charges are based on the compressed size of data.
Basically, unless you have flat-rate slot commitments, you are eligible for compressed storage pricing. In cases where you still have commitments running, you need to either cancel them or fully migrate to Editions. Otherwise, you can’t turn on Compressed Storage Pricing!
If you are on the Public Preview already, you will continue to be eligible.
How to activate the feature?
If you want to change to the compressed storage billing model, you can do that via API call, bq CLI command or plain SQL. The docs have you covered with examples, including a snippet to update all datasets within a project, for example this in SQL:
## update the dataset to physical storage billing model ALTER SCHEMA DATASET_NAME SET OPTIONS( storage_billing_model = ‘physical’);
However, you still need to reach out to your Google Sales representative or your DoiT account manager before activating the feature.
Two very different examples
Below you can find two examples of tables. The first table would benefit from a 93% decrease in storage, which would result in a 75% decrease in cost. In table 2 however, compressed storage would be a bad deal (+17500%).
Big Query Editions
We have a lengthy article that goes into more detail about the three newly launched Editions and can be found here.
Updates:
- April 6th: added eligibility criteria, added 7-day fail-safe period & corrected the GA date
