4.4 Measuring marketing value
One of the more difficult – and I’m sure frustrating – jobs of our recent time must be that of climate scientists. Attempting to explain to the general public how what we are observing in global climate studies deviates so far from the statistical norm without resorting to charts and data must challenge the patience of even the best. The situation is not helped, of course, by people who don’t want to know the data. Politicians, big businesses and celebrities who’ve made careers out of high-carbon activities don’t want to understand what damage is being done to the environment. You can’t sell to people that don’t want to buy.
Part of the difficulty in explaining climate change is that climate science is one of probability, not certainty. Sure, there is an average temperature for the month of June, but for how many hours on how many days is it actually that temperature? I can confidently say it’s a minority. Beyond this, however, Earth’s climate is complex anyway. With the exception of any readers who are members of the Flat Earth Society, we all know that the Earth rotates around the Sun. As it does so it wobbles leading to seasons of warm and cold as different parts of the globe receive more or less sunlight as a result. I’m talking about seasons, in case that wasn’t apparent. But the sun also changes. Every 11 years the Sun goes through a cycle of more or less activity. During this cycle the levels of sunspots, solar radiation and solar flares all increase and decrease dramatically. During a 28-year period in the late 1600s called the Maunder Minimum, an extended cycle of low sunspot activity occurred with fewer than 50 sunspots observed in a period that over a similar timespan today would expect to see 40,000–50,000. The Maunder Minimum coincided with a prolonged period of cool northern hemisphere temperatures knows as the Little Ice Age. Though it is not conclusive that solar activity caused this cooling, data does suggest that it solar cycles do impact the jet stream and does result in colder winters.
Then there are the Milankovitch cycles. What, you say, are Milankovitch cycles? Some kind of Russian pedal-drive transport? No, Milankovitch cycles are the complex interactions of between the planets and the Earth that pull it into more or less eccentric rotation around the sun. If the Earth rotates around the sun in a perfect circle then the solar energy the Earth receives is constant. But if the other planets tug on the Earth they cause it to move further or closer to the sun throughout it’s rotation. If the Earth has an eccentric rotation then during the summer it may pull closer to the sun and in the winter it will be further away resulting in warmer summers and colder winters from more or less solar energy being received. The alignment of the planets can have a marked impact of moving the Earth in its solar rotation and so on the climate of each season.
What does this have to do with measuring performance? Quite simply, marketing has the same problems as climate scientists, but rarely has the bravery to try to genuinely find out what is driving the change. Is it getting warmer because we’re moving from winter into spring? Or because we’re moving from a solar minimum to a solar maximum? Or because we’re pumping too much pollution into the atmosphere? Milankovitch cycles happen every 100,000 years, solar cycles every 11 years, seasons every year and day/night cycles every 24 hours. If you randomly stand outside your front door and try to measure global warming on a hand-held thermometer you’re not going to get a good conclusion.
Measurement needs to look at data and understand what is noise, what is cyclical or seasonal pattern and what is trend. Then it needs to understand what a deviation from trend. Climate change is a deviation from trend because nothing about recent history is close to the data we’re seeing, and data correlates with massive increase in carbon production. For what it’s worth, climate change from human activity is not a new thing – increased burning during the Roman Empire can also be catalogued in ice samples. The big difference today is the pace and scale of the change we are observing.
Did marketing campaigns drive an increase in brand perception or did some external factor influence it? Did marketing support pipeline creation with relevant opportunities, or would those opportunities have been created by sales anyway? Perhaps you might not care and might simply claim the win, take the bonus and move on. You may conclude that something Marketing did must have supported those goals being achieved, therefore Marketing did its job, but that’s not the same as presenting concrete-looking numbers in a QBR and making multi-million dollar decisions as a result.
Optimising performance
When you run digital campaigns you will have an ad driving to content. Does having similar messaging and creative in the ad and the content make a difference? Conceptually, of course it does. You want people to recognise that what they clicked on is what they’re receiving. But how much similarity or difference matters? Objectively, once they’ve clicked the ad and submitted their details, the ad has done its job. How would you look at the impact of copy that was similar or different? Are you best to optimise ad copy and then update the asset? Or optimise ad copy and not worry about the asset being slightly different? Or make them both the same?
One approach could be to pick some of your ads with copy different from the content and look at the ad performance. If you’re looking to show a correlation between disconnected messaging then you might start by looking at the lowest performing ad-content variations and then pull some examples from there. Conclusion drawn: when you have different copy you see poor performance. But what happens if you take some of the best performing ads and do the same check? Do you see all ads that have copy matching across the ad and content or do you see a similar pattern of matching and non-matching as low-performing ads? And what about taking the same approach looking at best and worst performing ads with copy that matches. Consider this: what happens if there isn’t a hard-and-fast rule? Is 10% better 30% of the time the type of improvement you want to focus effort on? What about 30% better 10% of the time?
A campaign performance of 10x ROI means very little in isolation. Similar to a 10 degree change in temperature – it means nothing. 10 degrees is not an uncommon daily temperature, or annual change if you look at daily averages across seasons. If the organisational benchmark for campaign performance is a 30x ROI and then $10m pipeline from $1m investment could widely be regarded as a failure. Similarly creating $100m of impact for $10m investment might also be deemed inefficient. All metrics need to be qualified against comparative benchmarks to have meaning. How much effort and budget was invested and how much did that deliver. Marketing organisations at a macro and functional level need to understand how success should be measured both in absolute (what value was created), baselined (how much investment was needed to generate value) and relative (how does the baselined value align to trend lines over time) terms before any performance reporting can be judged.
Quantum marketing measurement
As Eric Schmidt, cofounder or Google remarked: between the dawn of civilisation through 2003, 5 exabytes of information was created; that much information is now created every two days. Data is at the heart of many marketer’s professional lives: analysing it, reporting it, trying to predict it and frequently justifying or post-rationalising it. Greater access to data means a greater potential to learn. Or so it may appear.
Quantum mechanics tells us that at a certain scale it becomes impossible to know where something is and how much momentum it has beyond a level of precision. You can be perfectly confident in knowing where something is, but then you don’t know how much momentum it has. Or you can know its momentum but not be fully sure where it is. But you cannot know both. It also tells us that something can be in two places at once and only when you try to accurately measure it do you determine which one it actually is. The act of measuring changes the outcome.
The same is true in marketing. At the granular level it is easy to look at any piece of marketing creative and put data against how it performed. A 2% click-through rate. Or three leads generated. Or 300 downloads. But none of those data points represent that ad truly in isolation of all other activity. What about all the other ad formats a prospect may have seen before they clicked on this one. Or what about a piece of positive PR that they read before they converted to a lead? Or a word-of-mouth recommendation based upon an event someone had gone to, which resulted in a download. Yes, the data represents the performance of the ad, but not it doesn’t represent how well the ad alone performed.
Now imagine a highly competitive environment where targets must be met and proving performance is the path to promotion. The most positive data points are cherry picked and the rest are disregarded so that the story everyone hears is how well things are going. Everything in isolation appears excellent. But when everything is excellent, nothing can be improved or optimised. The reality of most modern marketing is that by striving for quantum-level data analysis the macro-level impact is not always met.
Quantum mechanics is essentially the study of probability. It is, indeed, possible for your to run straight through a wall. Statistically speaking every single atom of your body may miss ever single atom of a wall and you’ll pass to the other side to the amazement of all who behold it. But I don’t suggest trying it, because it’s not going to happen. In all probability, you’ll bounce straight off the wall to the much hilarity. Marketers can learn from this. Its fine to look at micro-level performance and work out if you change a CTA button from square corners to round corners but, really, look bigger picture. The only things you should care about are whether statistically speaking you’re making progress over time. Are you seeing improvement month over month, year over year? Consistently. Because if you are you can have confidence that what you’re doing is having an impact and you should continue to drive improvement. You’re standing on a floor that you won’t suddenly fall through because your luck has run out and whatever coincidence was artificially inflating your performance has stopped.