A/B Testing

Easily run automated tests from inside the platform

A/B Tests can be set up and run between any number of creatives within a campaign. Based on your goals, you can choose to optimize on overall conversion, CAC, or ROAS. Common test variables include:

• Size or Format: 4" x 6" vs 6" x 11" vs Letter

• Promotional Offer: e.g. "Free Shipping" vs "$10 OFF" vs "20% OFF"

• Postage: First Class vs. Standard

How to Setup:

One Time Send

If you are doing a one time send, you can randomize amongst all active creatives. To A/B test a One Time Send, select Multiple Creatives.

Please take care of any changes you make to your creatives within the campaign after you process the One Time Send. If any creatives are deleted or added within the next few hours as mailings are processing and going into production, the changes will be reflected in what mailers are ultimately sent out.

If you delete all the creative associated with a One Time Send while it is processing, the mailings that are not yet processed won't send.

Note on Billing

When you process a One Time Send that uses the Multiple Creatives A/B testing feature, you are billed according to your highest-priced, mailable creative.

• This will first draw from your credit balance.

• If your credit balance is insufficient, an additional charge will be made to the card on file to make up the difference.

At the time you process the mailing, we don't yet know the final distribution of addresses, because the system must first append or cleanse addresses and check your suppression settings before resulting in the final set of mailable addresses and randomizing creative assignment.

Once the creative assignment has finished, and final A/B testing ratios are confirmed, if the actual cost was lower than what was billed, the difference will return to your account's credit balance.

Shopify

Each Shopify trigger is connected to a campaign, which you specify when setting up the trigger.‌

• If you set a default creative for the campaign, the trigger will use the default.

• In the absence of a "Default", it will randomize amongst all other active creatives in the campaign.

To A/B test creatives in your Shopify trigger:‌

1. Make sure that there is no "Default" creative set.

2. Make sure you have one or more active creatives in the campaign.

Shopify triggers will always reflect the current creatives in your campaign.

For example, if you were randomizing amongst A and B, and you add a new creative C, your active Shopify trigger will start to randomize amongst A, B, and C.‌

If you were using A as a default and you change the default to C, the trigger will start to send out C.

Mailing API

Each time you create a mailing using the Poplar API, you can specify a creative_idin the payload.

• If you specify the creative_id, and the creative is active, the trigger will use that creative.

• If you don't specify a creative_id AND there is a default creative for the campaign, the trigger will use the default creative

• If you don't specify a creative_idAND no default is set for the campaign, the trigger will randomize amongst active creatives in that campaign.

How to Unset a "Default" Creative

If you have a default creative set, and you want to unmark it as "Default":

1. Deactivate the creative.

2. (Optional) Reactivate it if you still want it to be part of your A/B tests.

3. Once you do this, the creative will no longer be the default.

Analyzing Results

‌The mail file for the campaign will have a column with the creative ID, when running your own analysis you can use this to split out the different creatives and look at individual results. We can also run this as part of our complimentary matchback analysis on request.