One of the presentations I enjoyed the most at the recent 2011 Market Research Annual Conference in Washington DC was the one by Barry Blyn from ESPN. He provided superb examples of how market research should be implemented and add value to an organization.

First, Blyn made an important distinction between Measurement and Insights. Measurement tells you what people did, while Insights tells you why they did it and how to get them to act in the future.  In my view this is where market researchers can add most value.

ESPN’s research efforts are led by these guiding principles:

1. Research projects are about insights working more like weathervanes than speed cops. It is more important to know where the business is going (weathervane) than measuring how fast things went by (speed cop).
2. Research projects are led by neither quantitative nor qualitative work. They work together as peanut butter and jelly. Results from any research method are presented together to answer the business problem.

According to Blyn, ESPN has become fanatical about listening to their audience. In 3 years they have conducted more than 400 in-depth interviews all over the country and 15,000 surveys. In search for insights they have been combining traditional and innovative types of research with consumers trying to get at the heart of the business challenges ESPN faces. Blyn presented two examples of how these guidelines are implemented:

• Brand Health Tracking (Crusade): This is a brand tracking program which is seen as an orchestra where different research methods are thought of as instruments and are combined to reach the ultimate goal: monitor ESPSN’s brand strengths and hunt for weaknesses making sure ESPN is always in touch with fans’ perceptions about the ESPN brand. In this program, ESPN combines qualitative and quantitative methods such as:
  • RDD telephone interviews with 1,500 adults 12+
  • Quarterly online surveys with 1,600
  • Backyard barbecues with fathers and sons (group discussions)
  • Conflict groups with ESPN fans debating ESPN detractors
  • “College campus crawls” (Group discussions with students about sports over beers and wings at a college bar)
  • Word of mouth/social media  monitoring
  • Attending a baseball games with fans
  • Brand Eulogy (Imagine ESPN has passed away and deliver a eulogy)

• Deprivation study: This is by far the most provocative research approach Blyn presented. In this case, they paid $400 to 60 avid sport fans (30 who only used ESPN TV and 30 who use ESPN TV and other platforms to get sport news) for abstaining from watching ESPN during football season. The main objectives were to:
  • Get vivid illustrations of how consumers felt about the brand
  • Identify where fans were going when they couldn’t watch ESPN TV (identify content “blindspots”)
  • Build a five-tool player, meaning being a brand that can do it all for everybody in different platforms

In this study, ESPN allowed participants to provide feedback through different channels: video and audio journals, focus groups (“therapy” sessions), before and after surveys, and in-depth interviews. From this research it became clear that ESPN needed to align its different brand properties from a fan centric point of view.

In a time when many believe the market research industry have just missed the train, the ESPN case shows that to make research relevant to organizations today, market researchers need to:

1. Become more insight-centric. Bring the insights into the spotlight and leave the research methods used in the background
2. Provide forward-looking insights
3. Use hybrid research approaches and triangulate results

I think we can all do that.

There are many things to consider if we want to write surveys that gather high quality data, including data collection method, respondent effort requested, question wording, order, format, structure, visual layout behaviors to be measured, accuracy of the elicited information, among others. Although all these issues are important, at the end of the day, what we want is to create surveys that yield results that are valid and reliable.

Validity and reliability are often discussed in the field of psychometrics, but not so much in market research, although it is assumed they are present. 

Validity is concerned with the accuracy of our measurement, and it is often discussed in the context of sample representativeness. However, validity is also affected by survey design since it also depends on asking questions that measure what we are supposed to be measuring.

Most surveys often have what is called face validity, which is a matter of appearances. The questions seem like a reasonable way to obtain the information we are looking for, but are they really? There are other types of validity survey writers should strive for:

• Content validity: This is related to our ability to create questions that reflect the issue we are researching and make sure that key related subjects are not excluded. For example, if we are interested in learning how consumers use hair styling products, and only ask how they have used them in the past week, we are likely to miss information about how these products are used under different weather conditions (given that humidity can give you a bad hair day in a blink of an eye) and end up with an incomplete picture of consumers’ behavior in this category.

• Internal validity: This asks whether the questions we pose can really explain the outcome we want to research. In our hair styling product example, we need to ask questions that help us identify factors that influence the selection of hair styling products. Here we are looking for a relationship between independent variables (e.g. hair type, desired hair style etc.) and the dependent variable (e.g. likelihood to buy the hair styling products).

• External validity: This refers to the extend in which the results can be generalized to the target population the survey sample is representing. As we all know, the way we ask questions will determine the answer we get, so the questions should reflect how the target population talks and think about the issue under research, which often call for the need to conduct exploratory qualitative research. In our example, if we want to estimate the share of preference our hair styling product would get in the hair styling category, we need to include other brands that represent this category, otherwise we can’t extrapolate the results to the category as a whole.

Reliability, on the other hand, is concerned with the consistency of our measurement, that’s the degree to which the questions used in a survey elicit the same type of information each time they are used under the same conditions. This is particularly important in satisfaction and brand tracking studies, as changes in question wording and structure are likely to elicit different responses.

Reliability is also related to internal consistency, which refers to the degree different questions or statements measure the same characteristic. A practical application of this concept can be found in marketing segmentation studies that try to capture psychographics and construct behavioral or satisfaction segments by asking respondent to rate a list of statements using different rating scales (e.g. agreement/disagreement; likes/dislikes; excellent/poor, etc.). In our example, if we want to identify “lovers of styling products,” the statements used to describe such consumers should provide a consistent description of this group. This can be tested by using correlations, split sample comparisons or methods such as Cronbach’s Alpha.

Validity and reliability are not always aligned. Reliability is needed, but not sufficient to establish validity. We can get high reliability and low validity. This would happen when the wrong questions are asked over and over again, consistently yielding bad information. Also, if the results show large variation, they may be valid, but not reliable. So, don’t forget to think about reliability and validity when writing your next survey and strive for reliable and valid results.

Today, I did a great webinar in collaboration with Surveyanalytics.com. We had some technical issues, but at least that happened after the main part of the webinar was over. You can check the presentation slides posted at Slideshare.net below.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

by Michaela Mora

Posted on April 15, 2010

Brand tracking studies allow marketers to monitor the health of the brand and provide insights into the effectiveness of marketing programs implemented by the company.

WHAT SHOULD BE TRACKED?

Each brand faces different issues, which often required customized tracking surveys. Nonetheless, at Relevant Insights, we always recommend our clients to include measurements of awareness, usage, brand attitudes, perceptions, and purchase intent in brand tracking studies.

• Awareness: both recall and recognition measures should be collected. They are different indicators of the strength of the competition among brands in the minds of the consumers. A brand that first comes to mind in certain situations is more likely to be considered than one that is only recognized when it is prompted to the consumer.

• Usage: this can be measured through recency, frequency of usage, and total spending in the brand, and product category. These brand tracking measures, not only tell us about consumer shopping behavior and preferences, but also are indicators of market share and “share of wallet,” which is the amount of consumer spending a brand is capturing and has a direct impact on a company’s revenues and profits.

• Brand Attitudes and Perceptions: this is usually captured through questions related to brand image and associations that consumers develop as they experience the brand and are exposed to its positioning message through PR, advertising and promotional programs. Many brand associations are often beliefs about product-related attributes and benefits. However, brand associations also include non-product-related and symbolic benefits. Product and non-product associations, as well as those related to price and value are important sources of brand equity and should be part of brand tracking studies. Some brand associations are stronger than others, are more easily recalled and are enough appealing that they become an important factor in a consumer’s decision to buy a brand. Some brands may be perceived as unique, but without strong and favorable brand associations, uniqueness really doesn’t matter (Keller, Strategic Brand Management, 1998).

  Many times, attitudes towards a brand go beyond the product to include attitudes toward the company. We are all aware how Toyota’s image has been tarnished, not only by the car recall due to defective accelerator pedals in several car models, but more so by how the company failed to promptly notify car owners, ending up with a fine of $16 million imposed by the US government. A survey conducted by Consumer Reports in February 2010 reported that the number of Toyota owners who would buy another Toyota next time they are in the market for a new car has declined by 10 percentage points from a survey conducted in December 2009. Another before-and-after-the-recall study conducted by Lightspeed and Ad Age, also showed how the number of Toyota owners that consider the brand reliable has decreased significantly, indicating how fast a brand image can deteriorate when the company doesn’t react quickly enough.

• Purchase intent: measures of likelihood to buy a brand or switch to a competitor are also indicators of brand health and should be part of brand tracking studies, but these questions should be put in context regarding specific product or brand, reason for the purchase, time, channel, price and other relevant factors to the purchase decision, so they can be predictive of actual purchase behavior.

WHEN AND WHO TO TRACK?

Brand tracking studies usually involve collecting quantitative data from consumers on a regular basis. One way to do it is to continuously collect information, which allow us to control for unusual marketing activities, in the analysis, and provide a more representative picture of how the brand stands in consumers’ mind and against competitors. However, this type of brand tracking may not be feasible due to budget and resources constraints, and there are other ways to do it (monthly, quarterly, annually, etc.) that can be equally effective.

When determining the frequency of data collection in brand tracking studies, we recommend clients to consider:

• Frequency of product purchase: for example durable goods with long purchase cycles can be tracked less frequently.

• Marketing activity in the product category: a category where brands are constantly launching marketing programs and promotions should be monitor more often.

• Level of competition in product category: highly competitive product categories, where new products and competitors are constantly trying to break in, should be tracked regularly.

• Stability of brand associations: brands with an established image that don’t show appreciable changes over time, can afford a less frequent brand tracking.

Brand tracking studies are often conducted with current customers, but monitoring non-users of the brand can prove to be invaluable to the development of an acquisition and market penetration strategy in search for business growth.

HOW TO INTERPRET BRAND TRACKING MEASURES?

Given the comparative nature of brand tracking studies, brand tracking measures tend to stay the same over time. However, they should be revised from time to time to assess their reliability and sensibility. They may be stable over time and thus reflect stability of brand associations, but they can also be unable to capture important shifts in the market due to changes in sociodemographic trends, competitive landscape and economic macrotrends.

Another issue with brand tracking measures is defining what constitutes the desirable level of a particular metric. Is a 50% level awareness good enough? It depends. It is all relative to the product category and the competitive environment. In low involvement product categories and those with many competitors, it may be difficult to get very high levels of awareness and strong brand associations, so the benchmark for what it is a good level for a metric differs across industries and product categories.

Finally, each brand tracking study should be customized to capture the brand associations that contribute the most to brand equity and the marketing activities that are effective at strengthening it. The goal is to identify key drivers that have an impact on consumers’ brand choice and purchase behavior and develop marketing tactics that can lead to brand growth and sustainability.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.