Overview
Reliable delivery is a basic expectation of email services. The increasingly complex ecosystem has led to more frequent email bounces. When encountering various errors, properly handling failures and retrying are crucial to ensuring email deliverability.
Currently, the community lacks a comprehensive understanding of email retry strategies, and bridging this gap can enhance the reliability of email services.
We deployed RetryWatch to simulate various types of bounce errors returned by email receiving servers, covering the DNS, TCP, and SMTP communication stages, in order to provide the same methodology as our study to test whether your email service provider may have defects in its email retry implementation.
What is RetryWatch
RetryWatch is designed to act as an email receiving server and can simulate a variety of delivery failure scenarios. To facilitate testing, each scenario is implemented through a distinct subdomain and username under our controlled domain.
When a tested system sends an email to a specific email address, RetryWatch returns the corresponding error response, thereby triggering the target system's retry mechanism.
RetryWatch consists of four main functional components:
Baseline Module
Evaluate the default delivery behavior across IPv4/IPv6 and using different MX priority configurations, under normal reception conditions.
DNS Error Simulation
In cases where DNS cannot be correctly resolved due to DNS misconfiguration and resolution service failures
TCP Error Simulation
TCP connection failures caused by port configuration, security policy issues or network-related problems.
SMTP Error Simulation
Return error codes at some SMTP stages, proactively terminate the SMTP connection, or keep silent when receiving SMTP commands.
How It Works
We use a controlled authoritative name server and a designed mail receiving server. Using accounts normally registered with the email service providers under test, we send specially crafted emails to our mail server.
The recipient address, subject line, and message body of each email are deliberately structured to distinguish specific measurement tasks and to serve as unique identifiers.
At different stages of the delivery process, our mail server exhibits distinct behaviors that induce delivery failures, prompting the email service providers to invoke their retry and forwarding strategies.
We monitor and log the entire process by which each tested provider attempts to deliver emails to our server, and perform a comprehensive analysis of the collected data.
Test Cases
We construct recipients using different subdomains and strings. Each email's recipient is specially crafted so that our controlled receiving server can distinguish the mail provider, measurement project, and experiment-related parameters. The table below describes the various subdomains we used.
| Description | Type | Subdomains |
|---|---|---|
| Baseline | Single stack | tmail, tmail4e, tmail6e, tmail4edual, tmail6edual, tmail4, tmail6 |
| Dual stack | tmail4hdual, tmail6hdual, tmail64s, tmail64d, tmail64dual, tmail64h4l6, tmail64h6l4 | |
| DNS error | Base | nxdomain, refused, servfail, timeout, nonmx, nonda, nondaaaa |
| TCP error | Base | icmp-dest-unreach, tcp-rst, tcp-slient |
| SMTP error | NDR | helo-550, helo-552, helo-554, helo-421, helo-450, helo-451, helo-452, mail-550, mail-552, mail-554, mail-421, mail-450, mail-451, mail-452, rcpt-550, rcpt-552, rcpt-554, rcpt-421, rcpt-450, rcpt-451, rcpt-452, data-550, data-552, data-554, data-421, data-450, data-451, data-452 |
| Timeout | slient-helo, slient-mail, slient-rcpt, slient-data | |
| Disconnect | close-conn-helo, close-conn-mail, close-conn-rcpt, close-conn-data |
Ethical Considerations
Experimental Environment and Targets
For public email providers under test, we use only accounts that we registered ourselves. For open-source email software, we deploy and evaluate them on controlled servers within our laboratory environment. The recipient servers used to simulate various error conditions are also hosted in our lab. Throughout the entire study, no emails are sent to mailboxes or servers outside our control, ensuring that no external users are affected by our experiments.
SMTP Communication Safety
The error messages used in our experiments are all common responses observed during normal SMTP delivery, and do not contain any malicious content or malformed characters. This ensures that our testing does not negatively impact the tested email systems.
Testing Rate Regulation
We strictly regulate the testing rate to avoid imposing excessive load on target servers. Each test is conducted sequentially, only after the previous one is completed. The frequency and number of retry attempts are entirely determined by the tested email systems themselves; we passively observe their behavior without any active interference.
Test Your Email Service Provider
The public testing functionality of this study is still under development. Thank you for your patience and interest — please check back later.