How to ensure sample composition stability for higher-quality survey data

Measuring whether and how brand marketing changes people’s brand perception often remains more art than science. Tools to track brand perception can easily cost millions across multiple markets but often come with serious challenges in data quality. One of the areas that can greatly contribute to ambiguity in the data authenticity and accuracy is the stability of the sample composition. 

Maintaining stable sample composition through ex-ante audience quotas can be extremely difficult and expensive

Ensuring that participants surveyed in each wave are similar is one of the most complex challenges in generating reliable brand tracking data, preventing KPI changes from being driven by demographic shifts in the sample. 

To illustrate: imagine that 80% of young men planning to buy a new car are considering buying a sports car brand, while only 20% of the rest of the population would consider doing so. If in the first wave, there are 50 young men in an overall sample of 1,000 people, but in the second wave, there are 100 young men, then the reported level of purchase consideration for the sports car brand in the overall population increases by 3%, from 23% to 26%. This is not because the consideration level for the sports car brand changed among young men or the rest of the population, but simply because the second wave contained more young men.

If it were only one or two audience characteristics driving different brand perceptions and preferences, then holding them stable during data collection (or weighting the data accordingly after collection) would be relatively easy.

But in most categories, there are lots of audience characteristics that drive brand consideration, sometimes even dozens. For EV car brands, for example, purchase consideration may depend not only on age and gender but also on education, income, environmental consciousness, family status, work commute and many other factors. Attempting to hold all of these characteristics - and combinations thereof - stable can lead to hundreds or even thousands of individual quota cells, which makes data collection extremely complicated and expensive.

Combining answer quotas with Bayesian analytics

To avoid the impact of sample composition fluctuations on brand KPIs without creating hundreds of quota cells, we’ve developed a three-step process:

1. Step one

We’ve developed a technology that enables us to automatically set quotas on individual answer options to ensure that we always have sufficient responses for every relevant audience characteristic. For example, if we’re interested in people who have a high environmental consciousness and also have young children, we ask the relevant questions as often as required to be able to use them for further statistical analysis.

2. Step two

We then use multilevel regression and poststratification (MRP), a technique to estimate subgroup relationships and adjust them to match population demographics for better representativeness, to calculate the proportion of each possible audience combination within the overall population. This enables us to determine, for example, that 0.9% of the sample consists of high-income young men planning to purchase a car. We do this for thousands of possible audience combinations.

3. Step three

Finally, we use the estimates for each of these thousands of cells to weigh the results of the KPI estimates of subsequent waves accordingly. As the dataset grows over time, these estimates become more and more precise, and each subsequent wave is weighted according to those distributions.

The process typically relies on significantly larger sample sizes, with 50,000 respondents per brand annually, often 5 to 10 times larger than traditional methodologies. It is also very computationally intense, but it enables us to hold the sample composition much more stable than through any other methodology. 

Being able to eliminate sample composition effects ex-post solves another undesirable facet of quota-based sampling, namely that quotas that are overrepresented in panels often fill very quickly, whereas those that are not, take weeks or even months to fill. This leads to a very “spiky” fieldwork distribution, with lots of interviews completed in the first days of fieldwork, followed by a trickle of additional interviews over the subsequent weeks and months.

The challenge with such highly skewed fieldwork is that important information risks getting lost. Imagine a brand launching a Q4 holiday season marketing campaign in late October. In a quota-based sampling approach with quarterly data collection, most of the data collection for the quarter might have already been completed by mid-October, so the impact of the campaign will not be measured in that wave. By the time the next wave starts in January, the campaign could already be over.

Final thoughts

Ensuring sample composition stability over time is a complex but crucial part of ensuring that the brand trends you are monitoring each wave are due to real movements and not just noise in the dataset. Using a combination of answer quotas with Bayesian statistical modelling enables us to:

1 - Collect data steadily

Since we don’t rely on ex-ante audience quotas, we’re able to set a steady rhythm of data collection, with around 100 to 150 interviews per day.

2 - Collect much larger sample sizes

We collect approximately 50,000 respondents per brand annually, often 5 to 10 times larger than traditional methodologies. 

3 - Compute typically complex audience combinations

Our MRP modelling ensures that we produce reliable audience data, even for more complex combinations of characteristics, with low margins of error.