How budget strategy in Facebook affects instant donor conversion

Experiment ID: 10424

Metric Measured: Donations

Timeframe: 1/11/2019 - 2/11/2019

Experiment Summary

Recent experiments had resulted in a significant improvement in Buckner's active acquisition campaign known as "7 Scriptures." The greatest lift was the result of technical optimization of the Facebook ads. With this in mind, we wanted to take this one step further and continue our testing of the various advertising techniques available to us through Facebook.

One of the ways ads are shown is based upon the method of bidding. Historically, we had set daily spending limits for each audience which equally spread the spend across each one. We had the hypothesis that if we allowed Facebook's algorithm to determine the allocation of spend, we may see better results. To test this, we set the budget at the campaign level, using Facebook's Budget Optimization tool instead of the ad set level (setting a daily limit per ad set). All ads and targeting remained the same. 

At the time of the test, each campaign was spending the same amount per day ($83 and change). 

Research Question

Does Facebook's Budget Optimization tool create more opportunities to get instant donors than setting the budget at the ad set level? 

Design

Ad Set Level Optimization

Daily Budget Optimization

Results

   Treatment Name Samples Conversions Conversion Rate Relative Difference Confidence
C Ad Set Level Optimization 33,439 30 0.1% 0.0%
T1 Daily Budget Optimization 30,439 23 0.1% -15.8% 46.7%
   Total 63,878 53 0.1%

This experiment has a required sample size of 317,901 in order to be valid. Since the experiment had a total sample size of 63,878, and the level of confidence is not above 95%, the experiment results are not valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results with a 46.7% level of confidence:

T
C
A
0.0% in traffic
-15.8% decrease in conversion rate
0.0% in average gift
-15.8% decrease in revenue

Key Learnings

This test was not statistically valid, but the campaign that had daily limits set for each ad set did slightly increase instant donor conversion in the time span of the test. After further discussion, we may believe that setting daily limits for the Budget Optimization was not the best route since it still limited Facebook's algorithm to daily amounts. Instead, we are going to next test setting a campaign spending limit that allows for the media to be spent whenever the algorithm determines it is best. This would potentially allow for increased spending on days that show increased conversion opportunity.