Issue Explained:
In a recent case, we saw that a customer updated their Domain Name System (DNS) host and Mail Exchange (MX) records in order to fix an urgent issue where emails were not being delivered. The customer uses Google Workspace as their email provider and made an emergency DNS provider change to use Cloudflare only to discover the change was not as seamless as expected. Emails sent to the customer’s domain returned a “Message Not Delivered” notice with the error “5.1.2” to all email senders. We were ultimately able to determine the old DNS provider MX information was temporarily stored in Google’s Public DNS Cache and serving information about previous DNS lookups with the steps below. This issue can potentially impact any customer when updating their website’s DNS hosting provider or individual records published on DNS.
Troubleshooting steps:
The Google Admin Toolbox Check MX tool, a general troubleshooting tool to check the health of domain DNS records compared to Google’s recommended configurations, reported the customer’s domain to have Google MX records correctly configured. However, upon initial testing, we observed that messages sent to the customer’s environment were delivered to an SMTP server with an IP address owned by the previous DNS provider with no error on the sender’s side. To view this information, we used the Admin Console Email Log Search tool in a separate Google Workspace environment to send test messages to the customer’s email address and view the delivery results.
Our next step was to check the Google Workspace Admin Console, the interface to administer users and mail routing rules that shows the publicly-accessible MX Record state. We checked the MX records that Google Workspace had listed in the “Settings for Gmail — Setup” and found a single entry for the customer’s old DNS provider. The entry we located was in conflict with the MX records reported by the Google Admin Toolbox for the domain which signaled to DoiT Engineers that this was likely a caching issue.
Below is an example of the Gmail Setup page highlighting an MX record for another email provider which can cause email delivery issues.
Cached Outlook MX record showing in Gmail Setup section.
Resolution:
Our next step was to flush Google’s DNS public cached records, which can be performed on any domain (including domains not owned by you). DNS records are cached on non-authoritative DNS providers when the Time to Live (TTL) is set with a number of seconds the currently published records should remain valid, before any new updates to the DNS records are propagated from the authoritative DNS provider. In this case, our customer moved their DNS hosting provider to Cloudflare (the Authoritative DNS provider) and the MX record values were cached in Google’s non-authoritative Public DNS records. Flushing Google’s Public DNS forces the service to update records from authoritative DNS servers for the domain. No Google account is required to use this tool, and there is a built-in rate-limiting mechanism to prevent abuse.
When entering the domain name and Resource Record (RR) Type, you may notice that “MX” is not listed as a type in the drop-down menu. This is OK, the “MX” record type can be entered manually and the tool will allow the DNS cache to be flushed.
Resource Record types drop down does not include “MX” type.
MX record type manually entered with a successful flush.
After flushing the DNS Cache, the Google Workspace Setup section was updated to reflect Google’s recommended MX configuration. Additionally, we observed that emails sent during the time when the Message Not Delivered notifications were being kicked back to senders started being delivered to the customer’s Gmail accounts.
MX records showing correctly in Gmail Setup after flushing cache.
Other popular DNS services such as Cloudflare’s 1.1.1.1 also have their own Purge Cache tool which could be used depending on the customer’s chosen DNS provider. Flushing the public DNS cache may not always solve this type of problem as Google documents that DNS changes are processed and propagated within 48 hours, but sometimes it can take up to 72 hours. The propagation time for DNS records will depend on the domain hosting provider and the Time to Live (TTL) configured for each record type.
Outcome:
After using Google’s Flush Cache tool, the customer started receiving emails almost immediately, including messages that were previously not delivered. The attention and speed at which DoiT International was able to help the customer exemplify our support dedication and administrative experience to help organizations and was described as “Perfect Service”.
