Testing Principles

Welcome to Testing Principles

In this video, we’re going to cover the key principles behind testing. This is a theoretical training that will teach you how to think and give you mental models that you can use to make better decisions when you’re media buying. If you’re ever facing a big decision, or you’re feeling any kind of emotions about your ads, revisit this video.

</aside>

Theory Of Testing 🎓

As a quick introductory note, please keep in mind that the purpose of this video is to give you mental models and decision-making frameworks.

This video will teach you how to think about ads. If I give specific examples or scenarios, these are used as hypotheticals and are NOT intended as practical action steps. The practical action steps will come in the following videos.

However, I want you to understand these principles before launching your ads, so that you don’t make any critical mistakes during the launch process or in the period immediately following that.

With that said, let’s get into it.

Scientific Testing 🧪

Throughout our testing process, we want to act like a scientist.

We want to create and test logical hypotheses based on logical reasoning and avoid making decisions based on emotion and irrational hypotheses.

There are 7 key principles we need to cover to ensure you test correctly:

Problem-seeking bias
First-principles thinking
Deductive reasoning
The explore-exploit dilemma
Constructing a hypothesis
Controlled experimentation
Statistical significance

Let’s cover each of these principles and discuss how to follow them when managing your ad account. This is critically important to ensure you don’t make mistakes.

Then, we’ll get into the actual nuts and bolts of how to manage your account.

Problem-Seeking Bias 😩

One thing I have noticed with my clients is that they often believe there is a problem when there isn’t a problem. If your funnel is doing 7X cash ROI, just scale it.

Seriously, don’t overthink this. The only real problem is low return on adspend. If you have a low return on adspend, then we can drill deeper and figure out why.

If your ROAS is good, scale.

We’ll discuss this in statistical significance, but just because you haven’t closed any clients after a small amount of adspend (e.g. $1,000), or your cost per call is fluctuating each week, also doesn’t mean there’s a problem.

First Principles Thinking 🥇

Most people reason by analogy. This thought process goes something like this: “X problem is similar to Y problem. A is true about Y, therefore A is probably true about X.”

Analogical reasoning can be useful, but it can lead you astray too. Just because a solution is working for someone else, doesn’t mean it will necessarily work for you.

A helpful way to combat this is reasoning from first principles. To do this, you identify something that you know for sure to be true, then form a hypothesis around that.

Example:

First principle → I put $1 into ads and got $1.15 back.

First principle → The conversion rate on my VSL is 3%.

First principle → Industry-leading funnels get a 5-10X ROAS.

Analogy → Maybe my market thinks VSLs are scammy.

Analogy → Maybe my market is too busy to do sales calls.

Analogy → Jack Smith does group sales calls and it works better.

It is critical that you avoid reasoning like this or you will remain stuck in the position you’re in now. I see this kind of thinking ALL the time and it’s poison. Stop it.

Start with things you can verify with absolute certainty and then reason up from there.

Deductive Reasoning 🧠

Deductive reasoning is reasoning from the general to the specific. This is where we start with broad premises that we know to always be true, and then reason down to a specific conclusion.

Example:

Premise #1 → All humans are mortal
Premise #2 → Jack is a human