How we measure ROI, and why our number looks smaller than most

Every agency shows you ROI numbers. Almost no two agencies calculate them the same way, and most of the choices they make push the number up.

This page explains how we do it, step by step. It also explains where our method is weak, because it is weak in places, and you should know where. It's a long read. We would rather over-explain than ask you to trust us.

The short version

Why agency ROI numbers are hard to trust

An ROI number depends on four choices, and most case studies don't tell you which ones they made.

Revenue or profit? "We made you €600,000" sounds great. But if your margin is 40%, that's €240,000 in profit. Still good, but a very different number.

Which costs? Some agencies divide by ad spend only and leave out their own fee.

Compared to what? Sales went up after the agency started. Would they have gone up anyway? Most case studies can't say.

For how long? One month? A year? Customer lifetime? Each gives a different answer.

None of these choices is a lie. But when you pick the friendly option four times in a row, you end up with a number that has little to do with the money in your bank account.

What A/B testing does well

The hardest question in marketing is "compared to what?" A/B testing answers it better than almost anything else.

We split your visitors at random. Half see the old page, half see the new one, at the same time, in the same season, with the same ads running. If the new page earns more per visitor, the change caused it. Nothing else differed between the two groups.

This is the biggest strength of our method. But a test result is still a measurement, and measurements have noise. That's where it gets tricky.

How we run our tests

Here are our settings, in plain words.

• We decide how long a test will run before it starts, and we don't stop it early because it looks good. Stopping early on a lucky streak is one of the most common ways to fool yourself.
• We measure revenue per visitor, not only conversion rate.
• We call a winner at 80% confidence, testing in one direction (is the new version better?).
• Our tests are sized to catch a real lift 80% of the time.

Many people expect 95% confidence. We use 80% on purpose. Putting a change live that turns out to do nothing costs you very little. Waiting for 95% means far fewer tests per year, slower learning, and real wins left on the table. For deciding what to ship, 80% is a good trade.

It has a price, though. At 80% confidence, about one in five changes that do nothing at all will still pass as a winner. That comes with the setting. We think that's fine for deciding what goes live. We don't think it's fine for counting money. So we treat those as two separate questions.

Two reasons test wins look bigger than they are

1. Some wins are luck. As we said above, some changes that do nothing will pass the test. If we counted those at face value, we'd be billing you for noise.

2. Even real wins are measured too high. This one is less obvious.

A/B testing works the same way. We only count tests that cross the winning line, and a test is more likely to cross the line when luck pushed it up. So the winners, as a group, are measured too high. Statisticians call this the winner's curse. It happens to every agency that runs tests. Most don't mention it.

There is a third, smaller problem. When you add up ten wins, you assume they stack neatly. Sometimes they overlap. Two changes that both calm the same worry in a shopper's mind won't give you double the lift. We keep this small by never running two high-impact tests on the same page at the same time.

How we correct for it: the reality factor

We multiply the measured lift from all our wins by one number, which we call the reality factor. The right factor depends on an agency's win rate, meaning the share of all finished tests that came out as winners.

Remember that one in five useless changes will pass anyway. So if only 30% of tests win, about half of those wins are probably luck. If 60% win, most of them are real. A low win rate means a bigger cut.

We're a young agency, and we haven't finished enough tests yet to trust our own win rate. A win rate based on a small number of tests can swing a lot from luck alone. So until we have 200 finished tests, we use a starting factor of 0.60. In plain words: we count 60% of what our tests measured. That matches a 40% win rate, which is the middle of the table.

Once we pass 200 tests, we'll switch to the factor our own win rate gives us, publish it on this page, and update it every year. If our win rate turns out lower than 40%, the factor goes down, and so will the ROI we report.

The table itself comes from a statistical model, not from a direct measurement. It assumes real wins are about as big as the smallest lift our tests are built to catch. If real wins are usually smaller than that, the true factor is lower than the table says. Further down, we explain what we do to check our numbers, and what we don't do.

Wins fade, so we fade them

A change that wins in March won't be worth the same in December. Your traffic mix shifts, your site changes, competitors copy what works. So we reduce each win by 20% every quarter.

Take a win that measures €10,000 in extra revenue per month:

That adds up to €88,560 for the year, not €120,000. As a shortcut: monthly lift × 8.856 = the 12-month value after fading. That's about 74% of the flat number.

The 20% is our judgment, not something we measured for your store. Some wins fade faster and some barely fade at all. We use one number for everything so that nobody, including us, can adjust it to make a report look better.

From revenue to profit, and what we count as cost

We multiply the extra revenue by your store-wide contribution margin. That's what is left after cost of goods, shipping, payment fees, discounts and returns.

This is a simplification. A win that works by getting people to add one more item may carry a different margin than your average order. Tracking margin test by test would be more exact, but for most stores the data isn't reliable enough yet. We make one exception. If a test involves a discount, a free gift or a free shipping threshold, we subtract that cost directly, because we know exactly what it is.

On the cost side, we count everything you have paid us. Our fee already covers tools, design and development. It also covers every losing test and every flat test. We don't get to count only the months that went well.

The full calculation, with an example

Say a 12-month programme costs €60,000. Our winning tests measured a combined lift of €50,000 in extra revenue per month. Your margin is 40%.

A quick note on wording. ROI subtracts the cost first. If we divided profit by cost without subtracting, we'd get 1.8x. That's called a return multiple. Both are fine, as long as the label is right. A "3x return" is a 200% ROI, not 300%.

Now look at the same programme, counted three ways:

The same tests give three very different stories. The third one is our best estimate of what really happened, and it's the number we stand behind. It's also the one you'll actually feel in your bank account and your P&L.

What you see in your reports

Every month you get a report.

For each test

• What we tested and why
• The result: winner, loser or flat. You see all of them, not only the wins.
• The measured lift
• The 12-month value of that test, before and after our cuts
• What we learned

For the whole programme

• How many tests we ran and how many won
• Added profit in euros. This is the number that matters most.
• Payback period: the month where added profit passes the total fees you have paid
• Two ROI numbers. The unadjusted one, so you can compare us with other agencies like for like, and the adjusted one, which is the one we believe.

Every 90 days

• Which kinds of tests win most often
• Our projected lift next to your store's actual revenue per visitor
• An updated 12-month projection
• The plan for the next quarter

Where our method is still weak

• The reality factor is a model, and right now it's a starting value. It could be too kind to us or too harsh. We don't measure it directly, and we explain why in the next section.
• The 20% fade is an assumption.
• Store-wide margin is a shortcut.
• We don't measure overlap between wins directly. We never run two high-impact tests on the same page at the same time, so the overlap is small. It isn't zero, because two wins on different pages can still calm the same worry. The reality factor covers some of it, but not precisely.
• It's a projection. It estimates the next 12 months. It is not a bank statement.
• We leave some good things out. Losing tests save you from shipping changes that would have cost you money. Research teaches you things about your customers that help your ads and emails. Your traffic may grow beyond our forecast. All of that has value, but we can't measure it well, so we leave it out.

How we check ourselves, and where we stop

The cleanest way to check our numbers would be a holdback test. After a winner goes live, you keep showing the old version to a small group of visitors, say 10%, and compare again after a few months.

We don't do this, for three reasons.

It's technically hard. We would have to keep an old version of your store alive for a small group of visitors, for months. Every new test would then have to work on both versions of the store.

It gets less exact over time. People clear their cookies, switch from phone to laptop, and come back looking like new visitors. After a few months the holdback group is no longer clean, and the answer gets blurry. And because the group is small, it takes about three times as long as a normal test to show anything clear.

It costs you money. Everyone in the holdback group sees a page we believe is worse, for months.

So here is what we do instead.

We compare our claims with your real numbers. Every 90 days we take your store's real revenue per visitor. That's total revenue divided by total visits, straight from Shopify. We compare it with the same three months last year, and put it next to what our wins claim.

Say our adjusted numbers claim the live wins add about 8%. If your real revenue per visitor is up by about that much or more, the story holds together. If it's flat or down, either something else pulled it down, or our numbers are too high. Then we dig into which one it is, and we tell you what we find.

This check can't prove us right. Prices, seasons, new products and ads all move revenue per visitor too. One thing to watch for: when you spend more on ads, you bring in more first-time visitors, and they buy less often. That pulls revenue per visitor down even when our work is paying off. So where we can, we compare like with like, for example paid social visitors this year against paid social visitors last year. Think of it as a smoke alarm. It can't tell you everything is fine. It can tell you when something is clearly wrong.

We will set the reality factor from our own data. Once we have 200 finished tests, our own win rate replaces the starting value, and we update it every year after that.

We show the workings. Every number in your report can be traced back to a test result in your own testing tool and your own analytics. You can check our math.

This means our adjusted ROI stays an estimate. We think it's a fair one. We can't prove it's exact, and we won't pretend otherwise.

Questions to ask any agency, including us

We've answered all seven on this page. If another agency can't answer them, you can still look at their number. Just don't put much weight on it.