The Economic Impact of Email Testing 2023

is often overlooked in many organizations for many reasons, but we believe it can be the single most significant factor in down-funnel engagement. Pre-Send testing involves testing the experiences for the email program.

In many organizations, the website and UX are owned by many different groups. Still, if email marketing plays a massive part in revenue, it’s time for us to take a significant role. Some examples of pre-send testing are as follows:

• Location of email sign-up
• Data collected during and post-sign-up
• Preference center branding and experience
• Acquisition source spending and tracking
• Opt-Down and unsubscribe experience
• Frequency caps and holdout group testing

As email marketers, we need to think about the entire program, from beginning to end, and it starts
way before you send the first email.

involves testing elements and sub-elements that will optimize the open/read of the email itself. For example, a subject line test would be a pre-open test because depending on the sub-elements of the test itself will affect whether or not the subscriber decides to engage with that particular email. The subject line test is the main element, and the following could be sub-elements used to optimize the pre-open:

• Urgent vs. Curious
• Emoji vs. Non-Emoji
• Short vs. Long
• $ vs. no $
• Personalization vs. No Personalization.

There can be well over a dozen combinations of sub-elements in a subject line test alone. Some other main elements of a pre-open test include day of the week, pre-header, from name, domain segmentation, and the famous time of day.

testing is testing anything inside the email creative that will drive the subscriber to action.
Pre-click testing is fun to talk about, but a little more complicated to pull off. A solid hypothesis with
concrete and aligned KPI’s to track are essential to test things like:

• CTA’s
• Personalization (beginner to advanced)
• Imagery
• Dynamic content. (Images to messaging)
• Headers
• Content

Pre-click testing can take on many forms, and depending on the type of test; it can have tens of thousands of combinations.

testing involves the subscriber experience once they leave the inbox. As an email marketing program owner, you should have oversight and influence on the subscriber experience and if you don’t, demand it.

Some examples of post-click testing include things like landing pages, in-app experiences, and site navigation.

Essentially, test classifications are how we execute a test, while a test type is what we test. In this instance, context matters, and whether you have different names for tests or how they are executed, they are entirely other things.

The publishing industry competes heavily for inbox attention and revenue in the new digital age. So testing across all four test types is paramount to ongoing growth in the program. To illustrate how testing can impact a publishing organization, we are going to assume the following:

Publishing Company Assumption

• Your list has 2,000,000 subscribers.
• Your average unique open rate is 19%.
• Your average unique CTR is 4%.
• Your average unique CTOR (Click-To-Open-Rate) is 14.5%.
• The revenue per page view is .07.

A common pre-open test that most publishing organizations like to do, but don’t do it as often as they should, is subject-line testing. In the example below, you will see that over time that consistent subject-line experimentation can pay off.

Subject Line A: 19% open, CTR of 4%, and revenue of .07 per page view, generates an average of 2.7 pages per click. This would mean that the gross revenue of only having one subject line would be $2,872.

If we add in another SL to test against:

Subject Line B: 20% open, 4.25% CTR with 2.7 pages per click yields gross revenue of $3,213, which is 11.8% greater than the first SL.

The $341 difference might not seem that much, but multiply an 11.8% impact for the long tail, and pre-open testing looks a bit more interesting. Adding a 3rd of 4th subject line to the test is where things can take off in terms of content and downstream revenue. Suppose you are a publisher and are not testing multiple subject lines on every send. In that case, you might be missing out on revenue and/or engagement from every subscriber, especially since you are content and copy driven.

If we were to use the same programmatic assumptions above, a pre-click Button Copy impact test might result in the following way:

• Button Copy A: Read More
• Button Copy B: Show Me!
• Button Copy C: Take Me There!
• Button Copy D: Read On!

At iPost, we have seen a 1-4% variance increase when button copy is tested over a period of time. If a 1% variance increase happens, it will result in 20,000 more clicks, and the revenue numbers look like this:

2M subscribers with a 4% CTR = 80,000 clicks x .07 = $5,600
VS.
2M subscribers with a 5% CTR = 100,000 clicks x .07 = $7,000 (a 25% increase)

These assumptions above take into consideration that every click only goes to one page and abandons. The revenue numbers can increase significantly, even if the average number of page views per clicker remains flat.

Conclusion for publishers: There is no silver bullet when it comes to testing and revenue impact. You have to put in the work to get the desired outcome, but there is little doubt around how effort = reward.

As any restaurant operator or franchisee knows, loyal customers are the key to success, and these days loyalty to any brand is difficult to gain and retain. Critical areas of testing for the restaurant/franchisee should focus on Pre-Send, Pre-Open, and Pre-Click test types.

To illustrate how testing can impact a restaurant/franchisee organization, we will assume the following:

Restaurant Assumption

• You have 500,000 subscribers on your list
• Your average unique open rate is 18%
• Your average unique click-thru rate is 2.4%
• 35% of your list is enrolled in your loyalty/rewards program.
• Your Customer Lifetime Value for those in the loyalty program is
  $75 while those not in the loyalty program have a CLV of $40
• At sign-up, the only data you capture is name and email.
• You send one welcome email with a coupon and then start
  sending out regular marketing emails.

In this instance, a pre-open and pre-click test type are critical for the email program. Since the CLV of a loyalty/ rewards customer is nearly double, testing an onboarding series to get them to convert to the loyalty rewards program should be paramount. It is a fact that the average consumer today is a part of many different loyalty/ rewards programs, all with various structures and reward tiers, so your program needs to stand out even more to retain your CLV.

Using the assumptions above, 175,000 subscribers are a part of your loyalty/rewards program. If you were to test messaging and positioning around your loyalty/rewards program inside a series of onboarding emails and achieve a 15% increase in your loyalty program sign-ups, you would effectively gain 75,000 new members that contribute double the CLV. Even if you retain 50% of those new members within the first 24 months, the time and resource investment to execute and convert new loyalists to your program are astronomical.

Lastly, a pre-send test of capturing some additional information from the subscriber is vital. For example, you can try and capture dietary restriction data such as gluten intolerance or other allergies. Once you have this information, you could isolate those data points and begin pre-open and pre-click test types. By doing this, you can see if the use of that data would yield greater engagement and conversion to either dine or become a part of your loyalty/ rewards program.

Restaurant and franchise economic impact of testing can be substantial, but only if done systematically and regularly. The possibilities are endless.

Parts of the iPost team have spent a good portion of their careers working with or inside associations/non-profit email programs. These organizations provide tremendous opportunities for programmatic optimization, and it starts with testing.

To illustrate how testing can impact an association/non-profit organization, we will assume the following:

Association/Non-Profit Assumption

• You have 320,000 subscribers on your list.
• Your average open rate is 23%
• Your average click-thru rate is 4.90%
• 60% of your list is either paying members of the association or donors to the non-profit
• Your email program is monetized via in-email advertisements, membership dues, page-views on-site, supplemental products, or paid partnerships with other organizations. In other words, your email program is measured on the ability to influence and help convert subscribers.
• The current attribution model is that of last-click, and there are no plans to migrate to a time-decay, linear, or create a custom model.

One of the most critical test types for an association/non-profit is pre-send. The fluidity and transparency of the sign-up experience will set the tone for what is to come from the brand. Mimicking a retail sign up experience is no longer an accepted best practice because the association/non-profit has to humanize their message and brand promise quickly or run the risk of silent abandonment. People need to know who you are, what you stand for, and what to expect to give up their email address.

A typical tactical element that associations/non-profits like to follow when it comes to building out their subscriber list is that of Confirmed-Opt-In (COI). COI is when the marketer/organization sends a confirmation email to the subscribers’ email address, which contains a link that must be clicked to confirm consent for the organization to continue sending email. COI has also been called Double-Opt-In in some email circles. In this instance, the COI email would be considered a pre-send test type because if the subscriber does not take any action, no further emails can be sent. In our experience, typical COI rates can range from 45%-85%, and no matter where your organization is on that conversion scale, there is always room for improvement.

Imagine creating a pre-send test of the COI email, which results in a 10% increase in conversions. What is an email address worth to your organization, and how does getting more people to participate in it convert into revenue? A pre-send COI test can involve elements such as cadence, frequency, SL length, word count, and CTA buttons that all can have a substantial long-term impact on the revenue of the program. 10 iPost.com 415-382-4000

Post-click testing is also super crucial for associations/non-profits, especially when the essential CTA’s are that of membership or donating. Having award-winning creative, actionable SL’s with clear email CTA’s is the equivalent of running out gas in the last turn of a car race. You can get the subscriber to click, but you can’t them to convert unless you have the influence to test the post-click experience. Using the assumptions above, imagine the 15,680 people who click on your emails only to get to an unoptimized page with little to no information, and the conversion rate drops. What does it mean in terms of revenue to have the conversion rate improve by 2%? Context and experience matters in this uber-competitive world, so you, as the email marketer, need to ensure you have influence or ownership of the post-click experience, especially if success in your job depends on it.