Answering Burning Econometrics Questions with Dr Grace Kite

Data isn’t new to marketers; its use can be traced to the 1980's with the emergence of database marketing. But what econometrics can do – even compared to modern uses and applications of marketing data – easily supersedes it. That’s probably why in 2005, Sir Martin Sorell called it ‘the holy grail’ of advertising. The main function of econometrics is in aiding marketers to make better decisions through data modelling and analysis in order to predict how marketing activities will impact sales. It’s the framework for a model that can isolate just how impactful a TV ad was on sales - and even suggest how those sales would have held up in a world where that ad had never existed. That, of course, is only the beginning of what econometrics can do and understanding its far-reaching capabilities, even superficially, can help anyone with a stake in advertising effectiveness. 

To explain the basics of econometrics to those not in the know (which, let’s face it, is most of us), LBB reached out to Dr Grace Kite, founder and economist at magic numbers with over 20 years of experience who’s worked on over 120 econometric projects across the main advertising category. Dr Grace states that econometrics is “the most reliable method” of understanding the results of past marketing activity since it can bring together multiple factors in a single model to track and measure they affect the sales line. The model “can see that advertising is working, how many sales it’s added, and which channels are most effective as a result,” says Dr Grace. 

What econometrics offers – which is invaluable in today’s advertising landscape – is certainty. Why spend on X when its effects can’t be accurately captured and attributed to that spend? Well, econometrics has the answer because it “considers everything involved in the purchase decision, those earlier interventions in the purchase journey, like seeing a TV ad, for example,” Dr Grace explains. 

LBB sat down with Dr Grace Kite to get a primer on econometrics, how it’s different – and more precise – than attribution modelling, and to understand why it’s the foundation of effectiveness in advertising. 

LBB> In 2005, Sir Martin Sorrell called econometrics the ‘holy grail’ of advertising. When did econometrics first start getting applied within advertising specifically?

Grace> Hello, the ‘holy grail’ of advertising! Wow, if only us economists could market it so well! So, econometrics has been around for a long time. In the 80s and '90s, in the generation of econometricians before me, it was mostly FMCG businesses who wanted it, and it was actually carried out by ad agencies. It seems strange now to have the numbers people next to the poster and filmmaking people, but that was where it started. Like a lot of good ideas, it was thought up by creative people! Before that was before my time, but people used to use their judgement and make bets about what would work.

LBB> What are the main benefits of engaging with the principles and frameworks of econometrics? What are some of the outcomes that econometric models can help to predict?

Grace> It’s using data to understand the results of the marketing you’ve done in the past. Econometrics is the most reliable method for doing that – because it doesn’t just look at marketing and sales, but it looks at everything else that’s going on too, and untangles it all. At magic numbers we call what we do, which is econometrics, “business economics” – because we do what economists do everywhere, quantify things and use models to aid decision making. We just apply those tools to business and especially marketing.

We can use the outputs to predict a whole range of business KPIs, from signups to revenue to brand equity measures based on different scenarios e.g. changing marketing budgets, different economic scenarios. Almost every econometrics business has a tool for predicting the future!

LBB> Can you please explain, in brief, how regression analysis is used to find and describe relationships in data and why this is useful? 

Grace> Alright! Let’s talk about ice cream. We’re sort of in the midst of an Indian summer in London, so it feels appropriate. And to keep it simple, let’s just talk about this ice-cream brand that’s only sold in supermarkets. 

We’d start with a line that shows weekly sales of ice cream over time, and the question that econometrics sets out to answer is:

“Why does this line go up or down? Why does it wiggle from week to week? Why does it trend up or down?”

Then we’ll consider how the ups and downs and trends and wiggles in a graph of each driver, over time, match the ups and downs and trends and wiggles in sales. 

So, in this example, let’s say the ice cream sales are trending up from 2021 onwards. And it turns out that, when we plot out the number of stores it’s stocked in, that’s also trending up from 2021 onwards. Great, that’s a match! Get the ice cream in more stores, get more sales – makes a lot of sense! So, we’re starting to explain the pattern of sales.

What else can we see? Well, sales go up in the middle of the year, and down at the start and end of the year. Why might that be? If you’re in the northern hemisphere, which these guys are, it’s warmer in the middle months. And, yep! If we plot weekly average temperature next to our sales chart, we can see they go up and down together. Once we’ve built the whole model, that allows us to say ‘Ok, so if we see temperature go up by 1 degree across a week, then we see sales go up by 5,000 let’s say.

LBB> What are the main econometric techniques for identifying the causal impact of advertising on sales? 

Grace> So, the relationships we’ve found in the ice cream example are super intuitive, right? They fit with what we already know about how the world works. More places you’re selling ice cream means more sales, and hotter weather means more people buying ice cream.

What econometrics does is to make a model, an equation, that says sales is made up of these different factors. And if you keep adding the right (sensible, intuitive) factors to the model, the line on the chart that the model creates will match the sales line we’ve been trying to explain: The gaps between the sales that we know happened and our model disappear.

Advertising isn’t a big driver like weather or number of shops, so the effect isn’t usually super visible in the sales line. But the model, because it takes account of everything else, can see it. It can see that advertising is working, how many sales it’s added, and so which channels are most effective.

What data and econometrics do is reduce the risk, make advertising an educated bet, and help get the organisation comfortable with taking that bet. It solves the problem of sceptical finance people. Though, I don’t really think that sceptical finance people are a problem, businesses need them… But many of them have studied economics at uni so they get econometrics, and this method really works for them.

LBB> How is using econometrics to assess advertising effectiveness different to the way marketers/strategists are currently engaging with data?

Grace> I think the difference is in looking at all the different factors that impact a KPI like sales, which is what econometrics does. Something we hear a lot is, “Our competitor had a deal on when our TV campaign was on air, so we think the TV worked but we aren’t sure.”

Everyone suspects that the competitor promo stole that sales bump, but we need to be sure before airing again. TV isn’t cheap!

Econometrics untangles the TV from the competitor activity, so we can see what did what.

And it isn’t just for evaluating advertising. It untangles the effect of price, website glitches, availability and so on. All the main drivers, regardless of whether they are trackable or not.

LBB> How is econometrics modelling different attribution – a form of modelling commonly used by marketers?

Grace> There’s a big problem with attribution. The last click is only a small part of the purchase journey, so choosing this way limits your options.

People do a lot of research and see a lot of marketing before finally typing the name of the brand they’ve decided on into Google. Last click attribution can only see that last step, so it can never recommend earlier interventions to the purchase journey, even if they work brilliantly. 

And multi-touch attribution, which will be way less useful once Google finally goes ahead with its cookie-cutting plan, can see more of the sales journey – for those customers who allow you to track them across the internet. But still has no idea about marketing offline, or anything like the weather, or what’s happening in customers’ pockets, which can all totally distort the view you get on what your advertising budget is doing.

Using last click, and so making sub-optimal choices about which media channels to buy, means leaving 35% of possible sales achievable for your budget on the table.

So, it’s just something that everyone should be moving away from. Econometrics considers everything involved in the purchase decision, those earlier interventions in the purchase journey, like seeing a TV ad, for example, are accounted for.

LBB> What impact does the quality and volume of data have on econometric modelling? What are the limitations of econometrics applications in advertising? 

Grace> I’ll just say, we love data! We love working with it, and digging through it looking for all the interesting patterns in it.

Ultimately, the better the data, the better the insights can be. For our tools to find the proper relationship between X and Y, the effect that X going up has on Y, the data we use for X needs to properly show how X was changing.

That might mean that if we’re trying to model weekly sales, the TV data needs to be weekly as well. It might mean that if you’ve got a channel like Facebook or YouTube, where the buying type can change a lot, that we can see that halfway through the campaign changed from conversions to reach.

But we’ve worked with loads of clients who aren’t data nerds. One of our skills, because we work with it all, is that we can help the people we work with through the data process, and help get the good stuff out.

With econometrics you need to take into account everything that has a meaningful effect on sales. And that’s different for each business – it’s really important to have some discovery at the beginning of the study, not every provider does this but we really believe in talking to people from all the different departments asking them about all of their initiatives, successful and not so successful. You end up with a long list of things that affect sales.

There’s always price (this is always the most important thing driving sales, at the end of the day customers are always thinking “what do I get?”, “how much does it cost?”).

There’s always availability (things like shelf space and shops stocking the product, or the website being down). Changes in the product or range of products. External factors like the economy, or the weather, or the pandemic we all went through. Of course, marketing comes in many forms! Everything from price promotions to all the detailed ads people buy online. 

And then there’s the random weird things that each business has, like the time there was a fire in the warehouse and suddenly Germany couldn’t get any denim! That’s a real example actually, and the warehouse fire had a lot of knock-on effects. The obvious ones were loads of products being out of stock, but there were also all kinds of things that normally would’ve been done but which didn’t happen because the teams were slightly panicking trying to fix supply lines. 

By getting all the data, and hearing from lots of people about what happened, we can make sure that all the relationships we’ve found are legitimate.

LBB> How useful and accurate is econometrics in evaluating components of campaigns that are relatively small, like influencer marketing?

Grace> Not to sit on the fence, but it depends how small!

We’ve actually just put out a super simple online tool called the ‘Spend Selector’ that can help you determine the spend level you need for your business to be able to measure the marketing activity you’ve run.

So you can know before trying to get into econometrics, “can we actually measure this?”.

LBB> Finally, where would you advise people (especially creatives) to begin if they want to start engaging with econometrics to improve the effectiveness of their campaigns?

Grace> You want a provider that has roots in the longer history of econometrics, but that’s also using modern methods - technology for getting the data wrangled and into shape. APIs and code and AI and always-on desktop decision tools.

Be careful with providers that are very new to it.

To learn more about how agencies and brands can usher in a golden age of effectiveness, get your tickets for the IPA Effectiveness Conference 2024 – a hybrid conference where creatives, strategists, brand marketers, and agency leaders alike can get involved in the timely and timeless conversation about effectiveness. The conference will also see the launch of a publication called ‘Making Effectiveness Work’ which delves into the current methodologies of data measurement while advising how to navigate them.