Introducing a simple Tasks emulator that makes it easy to deploy and debug.
When developing an application that uses Cloud Tasks (let’s say a web app), you need to see what happens to your Tasks: How they are scheduled, delivered, and processed by a task handler. Yet, though one is often requested, Google offers no emulator for developing with Cloud Tasks, comparable to what if offers for Datastore or PubSub.
My new Python Cloud Tasks In-Process Emulator answers that need. If you’re impatient and want to use it now, just jump to the “Quick Start,” below.
Ways to develop for Cloud Tasks
You have four choices:
- Use the real Cloud Tasks service.
- Use a localhost-based emulator that exposes the same HTTP interface as Cloud Tasks.
- Just skip it: Do not bother handling the tasks. You can let task-creation task silently fail, or fire off tasks with no handler defined.
- Use an in-process emulator. Inject it for use in development, while an implementation based on Cloud Tasks in used in deployed systems.
I recommend:
- An in-process emulator for development
- Live Cloud Tasks for integration tests in the cloud
- Omitting Cloud Tasks from unit tests.
Let’s explore why.
The benefits of in-process
Debugging
In development, you want to know where each task is, and what is in it. When your emulator is in-process, you can do that with your debugger, pausing the queue and inspecting the values.
Simple is good
The code in this emulator is kept very simple, supporting only creation and handling of tasks. That is on purpose: A simpler codebase (here, under 100 lines) is better when you are debugging.
Also, with an alpha product — and all Cloud Tasks emulators, including this one, are in alpha — you want the confidence of keeping all code in your direct control, so you can quickly debug and even edit it.
For fuller functionality and more realistic testing, I would use the real Cloud Tasks, in a cloud-deployed application.
Ease of use
Other Cloud Tasks emulators, like Potato London’s gcloud-tasks-emulator and Aert van de Hulsbeek’s cloud-tasks-emulator, run in localhost and have the advantage of exposing the same API as the cloud service; you need change only the endpoint. Also, code in any language can call these emulators; that is not possible with the in-process emulator.
On the other hand, they impose the additional burden of another process: launching it before each development session and keeping track of whether it has crashed or frozen. And though you can debug them if you launch them in a debugger, this is likely not a part of your main development process.
Don’t lose your Tasks
You have the choice of firing off tasks into Cloud Tasks, and then never handling them; or just failing silently to create them. This works well if your primary development flow does not involve handling the tasks: For example, if they are handled in a separate microservice that another team is responsible for. It also works well for unit tests, as these are generally not used for testing distributed asynchronous processes; that is the role of integration tests.
But it you want to see what actually happens with your tasks, and how the callback processes them, you need some sort of emulator.
Develop with the Cloud, or without it
For a full end-to-end-flow, you can also use the real Cloud Tasks in your development machine, getting the same API and functionality as in the deployed system. This works well for integration tests that are run in the cloud.
However, if you try this in your development machine, you will need to always have a network connection; and you will need to set up a proxy like ngrok to let Google Cloud Platform reach your local machine (as described here.) Using the real Cloud Tasks API also requires real queues, which means you either need a a Google Cloud project of your own, or else manage the queues so that each developer gets their own set.
It also means that debugging even a simple process on your development machine sends you to the Cloud Console for more information. For my purposes, I’d rather skip the setup and keep the debugging inside my process.
Features and limitations
This project supports the functionality that you typically use in development: Creating a task, and then handling the task in a callback.
However, because of the preference for simplicity mentioned above, it does not support other features, such as:
- Queue management, including creating, deleting, purging, pausing and resuming queues. (When you create a task, the queue is automatically created, if not yet present.)
- Rate-limits and retries.
- Deduplication and deletion of tasks.
If you would like new features (and if they are simple enough to be worth the added complexity), please submit Pull Requests or Issues at GitHub.
Quick Start
To use this emulator, just copy emulator.py into your codebase — you could even just copy/paste the code of that file straight from GitHub — and you are ready to go.
Create an Emulator object and pass a callback function, which will receive the payloads of the tasks that you create. Then, to send tasks, call the method Emulator.create_task. You can choose the name of the queue, as well as the scheduled delivery time.
Usage Example
In the project, you can see emulator.py at work inside a simple webapp. Start it by running local_server.py . Then browse to https://127.0.0.1:8080 (or just click the link that you will see in the console) and a task will be created. The task will be handled (printed to standard output), on schedule, three seconds later.
If you’re a stickler for code hygiene and want to keep the production codebase free of the emulator — omitting emulator.py in deployment — this example shows you how. In local_server.py, used in development, we inject an Emulator. In contrast , in a deployed server, where no such Emulator is injected, a new CloudTasksAccessor is created that invokes the real Cloud Tasks API, so keeping the server code (main.py) clear of any emulator code.
