Tidy up the resources cluttering in your dev and QA projects

As you design and develop new infrastructure on Google Cloud, you try out all kinds of resources. This leaves a mess in the project; and it also costs money as you leave these resources running.

You can delete resources by hand, but it can be hard to find everything across the many services in GCP: Compute, Containers, AppEngine, Cloud SQL, and more.

I created the open-source Safe Scrub to make this easy — and safe!

Safety First

Safe Scrub was designed to be safe.

Safe Scrub does not delete resources; rather, it generates a script that can delete resources.

Transparent Deletion Script

This deletion script is transparent — because it is simply a list of gcloud delete statements, you can easily understand what it is going to do to your resources. Once you generate this script, review every line to see exactly what is going be deleted, and only then run it.

That is the most important safety feature, but there are more.

Explicit Account and Project

Safe Scrub requires a JSON key file with credentials for a service account, rather than your logged-in user account. This is designed to require the conscious choice of a role; the service account should have a role with no write capabilities, like Project Viewer, as the script-generation script does not need or use write capabilities.

Beyond that, you could give a more limited role if you only want to list resources of certain types. There is no problem with failing on permissions. If the role is too restrictive (and also if a GCP API is not yet enabled), Safe Scrub continues with no problems, other than not being able to list the unavailable resource. It just emits an error message and keeps going.

Unlike gcloud itself, Safe Scrub requires you to explicitly state a project, to avoid using a default project and so accidentally listing resources that you did not intend to delete. The generated deletion script also specifies the project for every deletion command, so that deletion is not accidentally run against your current default project.

Filtering

Safe Scrub supports two kinds of filtering.

First, there is a command-line option so you can choose just the resources you want. You can filter by label, name, creation date, and much more, using the full power of gcloud filtering. (For Cloud Storage buckets, where gsutil rather than gcloud is used, only simple single-key label equality filters (key=value1) are supported. Otherwise, the filter is ignored.)

Second, Safe Scrub supports a no-deletion list in exclusions.txt. Resources that include any of these strings in their URI will not be included in the deletion script. To use this feature, run Safe Scrub, note items that should not be deleted in future, and add the given URI or an identifying part of the URI to exclusions.txt. (For Cloud Functions, the name rather than the URI is used, because of a bug in gcloud.)

For example, when these lines appear:

networks/default
firewalls/default

the default networks, and alsothe default firewalls (whose URIs have the form https://www.googleapis.com/compute/v1/projects/my-project/global/firewalls/default-allow-https ) and will not be deleted.

Monitoring the Deletion Process

When you first run your deletion scripts, you will want to watch the process as it happens. Safe Scrub prepends set -x to the deletion script so that you get debug output.

For easier monitoring during deletion, Safe Scrub generates an ordinary sequential script. However, you can also set it, using the -b switch, to generate background (asynchronous) commands, so that the deletion of multiple resources will happen concurrently. This is faster, yet harder to track, and it risks overloading systems with too many concurrent calls.

Next steps: Review and Run

That first step just gives you a deletion script. Your next step is to review it and remove lines for any resources that you want to keep. You can also use this opportunity to add lines to exclusions.txt for future filtering.

Actually deleting resources

When you are ready to delete, run the deletion script under an account that has the relevant write permissions, like Project Editor.

Dangerous mode

We love automation in the cloud, but when the mass deletion is involved, you should be extra-careful and track every step manually. Still, with a development or QA project, you might refine your Safe Scrub configuration enough that you feel comfortable generating and then running the deletion script automatically, in one step. To do that, just pipe output to bash, as illustrated in in dangerous-usage-example.sh. In this case, your service account should have read and write permissions, as for example Project Editor.

Use cases

Dev and QA vs. Production

Safe Scrub is intended for development and QA projects, where you want to start fresh at the end of the day or before a new test run. It is unlikely to be useful for production projects, where you should determine the potential dependencies between components before deleting anything.

Unblocking dependencies

It is also useful when a stray dependency is blocking the deletion of some other resource. For example, you cannot delete a Google Kubernetes Engine cluster if there is a backend pointing to it — even if the backend is not actually in use by a load balancer. Use Safe Scrub to produce deletion commands for all resources, then choose the command line need to delete the backend and release the lock on that cluster.

Current Scope and Future Possibilities

Supported and Unsupported Services

Google Cloud has over 90 products, and Safe Scrub does not cover them all. I focused on the common important services that are set up and torn down in typical development and QA. This includes resource types from GCE, PubSub, App Engine, Cloud SQL, Cloud Storage, Functions, and GKE. Among the services that are not supported are Composer, Data Catalog, Data Proc, and many more. The full list of supported and unsupported services appears in the usage text. (Run ./generate-deletion-script.sh .)

Supported and Unsupported Resource Types

Not necessarily all resource types in each service are supported, although I think I have covered the important ones, especially omitting the resource types that are very risky to delete. For example, in App Engine, services, versions, instances, and firewall rules are supported, but SSL certificates are not. If you want more APIs or resource types, please submit a pull request or add an issue at GitHub.

Future Implementation Options

A future version of Safe Scrub may use the command gcloud resources list(in alpha as of June 2020) to fully capture all resources. Still, it is not clear that we will go that route: Some types of resources are not amenable to deletion (like IAM, Secrets, and SSL Certificates, where you might delete something important). Also, some types of resources have different deletion commands. (For example, we use gsutil instead of gcloud for Cloud Storage, and will use bq for BigQuery.)

Getting Started

Click here to download the open-source Safe Scrub.

Unzip and run ./generate-deletion-script.sh to see usage instructions; file usage-example.sh shows an example.

The very simplest quick-start is this:

• Download a key for a service account with Project Viewer role, saving it as project-viewer-credentials.json in the Safe Script directory.
• Then run ./generate-deletion-script.sh -p <YOUR_PROJECT_NAME>.