Not long ago I got a call from a potential client asking for research to determine why a recent marketing campaign failed to increase sales, despite a significant increase in awareness.

He had conducted an advertising awareness survey, and the results showed that many in the target audience had noted the advertising and gave it high ratings, but didn’t make a purchase. All possible explanations were merely speculations. He couldn't pintpoint to any particular cause for this. The main problem was that he wanted to obtain evidence of a cause-effect relationship, but the research design was not appropriate for that.

The main method for causal research is experimentation. In experimental-based research, the causal or independent variables are manipulated in a relatively controlled environment. This means that other variables that may affect the dependent variable (e.g. sales) are controlled and monitored as much as possible.

In this case, the client had conducted the survey and analyzed the data without taking into account the effectiveness of the different marketing collaterals used, its market penetration, competitor activity, and some characteristics of the purchase decision makers. After doing some digging around, we uncovered that in some markets, competitors had launched high frequency advertising campaigns which helped the client indirectly by increasing category awareness, but not his sales. Also, the program targeted customers who were recent buyers and probably didn’t have a need for his products at that particular moment.

Surveys that are not designed as part of an experimental approach may show correlations, but not causality. To really connect the dots between cause and effect, we needed to create an experiment including different renditions of the marketing collaterals, different markets, customers at different stages in the purchase cycle, and different actions taken by competitors.

Experimentation in marketing has traditionally taken the form of standard test markets, in which test market are selected, controlled advertising is put in place, and the product is sold through regular distribution channels. The drawbacks of this type of test are that they can be time consuming, are often expensive and may be difficult to administer.

A more palatable solution is what is called simulated test markets in which individual are selected, exposed to the product or concept (e.g. via actual marketing collaterals), given the opportunity to buy the product in real life and if they buy it, they are asked to evaluate the product and state their repeat purchase intent. The trial and repeat estimates are combined with data about promotions, distribution levels, competitor activity and other relevant pieces of information.

Another possibility can be found in the popular freemium model, adopted by many businesses both in B2B and B2C, which mimics this process to some extent. This model can be used for experimentation and as a rich mine for insights at a low cost. The basic principle is to let people try it and observe what decision they make, after which research can follow to understand what drove their decision, controlling for other variables that may be affecting the outcome.

In short, if you want to understand cause and effect, you need to conduct experiments, which may include surveys as data collection method, but surveys in themselves can’t provide the answer. It is the  experimental design what will lead you to it.