Multiple-choice questions (check all that apply) are one of the most common question formats found in online surveys. However, there are a couple of problems with this type of question:

• It often makes it easy for respondents to engage in satisficing behavior, which occurs when respondents select the answer options without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement, rather than work on finding the optimal answers that best represent their opinion.

• We really don’t know what it means when an item from the list is not chosen. This could happen (Sudman and Bradburn, 1982) because:

1. The option didn’t apply to the respondent
2. The respondent is neutral or undecided
3. The respondent overlooked the item

 WHAT CAN WE DO ABOUT IT?

A solution to this problem is to ask multiple-choice questions as a series of forced yes/no answers for each of the question items. This format requires that respondents report a judgment on each of the items. Research has shown that forced yes/no questions encourage deeper processing time and discourage satisficing response strategies as measured by the time spent on answering forced yes/no vs. check-all questions and the number of items marked affirmatively in each question format. Research by Smyth et al. (2003), comparing results from both types of formats in online surveys has found that:

• Respondents who answered forced yes/no questions spent significantly more time responding than did respondents to the check-all formatted questions.

• The forced yes/no format yielded more options marked affirmatively than the check-all format.

We can argue that the longer time spent answering the forced yes/no questions is a mechanical function of the fact that respondents are forced to give an answer for each item and spend extra time marking “no,” which is not required in the check-all question.

However, the positive correlation between time spent on answering the question and the number of options selected has also been shown to be an indicator of deeper processing and more thoughtful answers for the check-all formatted questions as well. Respondents who spend more time answering check-all questions mark significantly more answers than those who answer check-all questions in less time.

Another research result supporting the hypothesis of deep processing is that no significant differences have been found in the number of options marked affirmatively between respondents that take longer time answering yes/no questions and check-all questions.

The yes/no format for multiple-choice question are not a 100% foolproof, as some respondent may still show satisficing behavior by marking yes or no for all options or marking them randomly. In this case we need to put quality checks in place during programming that take into account the time spent on the question and any straightlining patterns.

An issue that we also need to manage is the fact that sometimes respondents are undecided or think an option doesn’t apply to them. In this case it would be wrong to force them to give a yes or no answer. The best remedies against this problem are respondent screening and survey skips that would avoid showing options that don’t apply. In cases in which there is still room for this problem, I recommend adding a third “Don’t Know/Not Applicable” option.

IMPLICATIONS FOR MIXED DATA COLLECTION MODES

 The forced yes/no format for multiple-choice questions is commonly used in phone surveys since it is impractical to read all the options to respondents and expect them to remember them all to answer the question. Often, when mixed data collection modes are used, (phone/online, phone/paper), the yes/no and check-all question formats are treated as equivalent, assuming they are answered the same way. Research suggests that this would be a mistake.

Experiments carried out by Smyth et al. (2008) with phone and online survey using both question formats have shown that the forced yes/no format yields consistently more options marked affirmatively than check-all formatted questions in online self-administrated and phone-administrated surveys. This supports the idea that results from both question formats are not comparable and shouldn’t be treated interchangeably.

KEY TAKEAWAYS

• You are better off using forced yes/no format for multiple-choice questions in order to elicit deeper processing and minimize satisficing behaviors.

• Do not mix the yes/no and check-all formats across data collection modes, as results are not comparable.

Concept testing is one of the most widely used research techniques used in new product development. I often meet clients wanting to test new product concepts, who get surprised when we discuss all the issues we need to consider in concept testing.

In a recent article by Jerry Thomas, from Decision Analyst, discusses what he has seen to work best in concept testing, which has also been my experience. Thomas calls for creating a research system supported by standards and standardized processes to make sure all new product concepts are tested the same way inside a company, which allows for comparisons of concept tests over time and across concepts. According to Thomas, concept standards should address at least the following:

• Format: Print or video
• Presence of illustrations
• Content of illustrations: Type and size of illustrations (include retail package or not?)
• Copy: Style, tone, complexity and length
• Font types and sizes
• Degree of finish (rough or magazine-ready?)
• Presence of price (priced or unpriced?)
• Presence of brand (branded or unbranded?)

Unfortunately, in my experience, most companies don’t have such a system and treat concept tests as isolated ad-hoc projects. This is also reflected in the quality of the concept descriptions, which differ in style, content richness and format across concepts. Concepts are often poorly written by internal research of marketing staff and have little resemble to how the product will actually be positioned and presented to consumers.

Thomas also discussed some of the decisions we need to make, when designing concept tests. These decisions include: which approach to use (monadic vs. multiple, decision often driven by cost), whether to show prices and brands, which sampling strategy to follow, which normative data to use as a reference (if available), and what analytical approach to adopt.

As Thomas indicates about “one-in-10 concepts will be good enough to warrant investments in product development to create the product that fulfills the promise of the concept. And roughly one-in-five of this group will eventually be deemed worthy of taking to market. It’s a numbers game, and the odds are against you.” Companies that establish research systems to support consistent concept testing and chose to follow best practices in the issues discussed above, are more likely to develop successful new products.

To learn more about our Product Concept Testing service visit Concept Testing and Product Optimization.

 We, as consumers, don’t always do what we say. This is a fact that market researchers have to wrestle with in the design phase of any research project. As Jeffrey Henning pointed out in his recent article, Respondents as Economic Actors: Behavioral Economics & MR, some schools of economic thought have been drawing attention to the inadequacy of the rational choice theory, which assumes that consumers have perfect information about all the alternatives and weigh in pros and cons before making a purchase decision.

Although sometimes this may be true for certain purchase decisions in product categories a consumer values and is engaged with for a variety of reasons, many times consumers make decisions with incomplete information or simply skip the evaluation of options and make impulse purchases or go with what is available.

Henning summarizes the situations where, as behavioral economists have pointed out, actual behavior doesn’t match the rational approach to decision making. These are:

• Rules of thumb: Here consumers rely on heuristics or rules of thumb as shortcuts to decision making. They don’t take into account all possible options and may not make optimal decisions, but their decisions are “good enough.” This may be due to information overload, too many options, time constraints, financial situation, and lack of category involvement, among other factors.

• Emotional arousal: Consumers sometimes make purchase decisions influenced by their emotional state. If the person is calm she is more likely to think through her purchase decision. However, if the person is experiencing strong emotions (positive or negative), they are more likely to succumb to impulse purchases without much thought of the long term consequences.

• Framing: The context often influences purchase decisions in ways consumers may not be aware of. We often compare products to others that are present, particularly on price. The same product may look like a good value at one store or terribly expensive at another depending on competing alternatives and our expectations. Store atmosphere, layout, music, scents, in-store advertising can invite or discourage consumers to buy. For online retailers, the website design, layout, navigation path, graphics, type and amount of information, and trustworthiness indicators, among others, provide a context that influence our decision to buy from a particular online retailer.

• Cognitive biases: As Henning puts it, “individuals overvalue items they own (the endowment effect) or have invested in (the sunk-cost fallacy),” and tend to feel losses more intensely than gains (this may explain why for some, paying for shipping feels worse even if the cost of shipping may be compensated by a price discount). We often assume that others think like us, but are also influenced by the decisions of others (e.g. recommendations by word-of-mouth). We also seem to be wired to think short-term and have a hard time resisting instant gratification, which may interfere with rational decisions that would be more beneficial to us in the long run.

We can probably find these situations in many categories and some are likely to be more prevalent than others. In my opinion, in order to tackle this problem from a research perspective, we first need to understand how consumers make purchase decisions in a particular product category and identify potential segments with different decision making approaches.

For instance, a consumer may consider clothing detergent a commodity and buys whatever brand is on sale at the time of purchase, while another browsers the detergent aisle, opening bottles to check for fragrance, and reading packaging labels searching for harmful ingredients for herself or the environment. The key is segmentation within product categories based on purchase decision approaches.

To capture the nuances and situations influencing purchase decisions, we can’t rely only on traditional concept tests or focus groups. These need to be combined with methods that go deeper and allow us to understand consumer emotions, purchase context, cognitive biases and rules of thumbs. Some of the research techniques that are useful for these purposes are:

• Adaptive Choice-Based Conjoint analysis: Consumers are asked to build their own product based on a set of criteria, information that is used to understand the rules they use to choose products (must-haves and unacceptables) and to present relevant alternatives they would actually purchase.
• Shop-alongs: We go along with consumers in their shopping trip and observe how they make purchase decisions, what the motivators are, how the context influence their decision, the role of emotions, etc.
• Mystery shopping: Consumers get immersed in a shopping occasion and report back their personal experience with different aspects of the purchase occasion.
• Journaling about experiences: Consumers report about their experience with products and services in a journal format using text, video or pictures as the experience progresses.
• Ethnographic interviews: Consumers are interviewed as they carry on different tasks or use products in real time and environment.
• Mobile surveys in real time: Consumers are asked about their immediate and current experience, feelings and opinions via text messages.
• On-site observation: Acting like a fly on the wall, we can watch how consumers buy and use the products and integrate them in their daily life.
• In-Depth interviews: We delve deeper into purchase drivers, cognitive biases, situational factors, etc. Projective techniques can be used to uncover motivators not consciously recognized.
• Neuromarketing research: We use neuroscience, psychology and other cognitive science techniques to study consumer responses to marketing stimuli and products. Some of the responses measured include eye tracking, heart rate, electroencephalography – EEG, functional magnetic resonance imaging – fMRI, galvonic skin responses, etc.

If you want to understand the gap between what consumers do and say, don’t rely only on one research methodology, as each research method provides data that reflect only a few facets of the consumer.

The market research field is going through changes driven in part by the advent of new data collection technologies, increased competition in almost any product category, and research clients’ limited resources in terms of budget and staff.

In my opinion, many of these changes are here to stay and will continue to be “hot” trends to consider in years to come.

TOP 5 HOT, LIVING MARKET RESEARCH TRENDS

• Democratization of research:  Thanks to cheap online data collection tools for both quantitative and qualitative research more companies are doing surveys and using online tools for qualitative research. This has also accelerated the proliferation of DIY research. DIY is here to stay, but vendors will still be needed to provide expertise and objective insights.

• Commoditization of research: In an effort to save money, many companies have laid off many experienced (and more expensive) market researchers, and replace them with DIY research teams with less experienced staff.  It seems many companies are resigned to get “good enough” research as long as it is fast and cheap. Unfortunately, this attitude combined with access to inexpensive data collection tools has often a negative impact on the quality of research. Many companies that still use research vendors, are creating smaller preferred research vendor lists (with the risk of overpaying for services due to lack of competition) and are handing vendor selection to Procurement departments, which have a hard time realizing research services are not widgets. On the agency side, off-shoring of many tasks of the research process that have become commodities will continue to grow for cost saving purposes.

• Use of market segmentation research: More and more companies are realizing that the “one-size” fits all approach doesn’t work in today’s market place where customers have so many options available, so market segmentation research will continue to be of great interest to marketers and research clients to increase marketing strategy effectiveness.

• Use of branding research : Like market segmentation, branding is top of mind for companies looking for differentiation. Companies need to understand sources of brand equity in order to find a brand positioning that resonate with target audiences.

• Increased use of new online and mobile qualitative research techniques: Qualitative research has seen an explosion of online data collection tools that allows capturing data faster, cheaper and in large quantities, breaking geographic barriers and giving access to hard-to-get groups and real-time consumption occasions. These techniques have revitalized the qualitative research field and their use will continue to grow.

On the other hand, some practices are bound to die with time as they don’t add values to research users or new affordable research alternatives become available.

TOP 5 NOT-SO-HOT, DYING MARKET RESEARCH TRENDS

• Rely only on one data collection methodology: Almost all sample sources suffer from coverage bias nowadays. We know we can’t reach all groups online and more and more households are dropping their LAN lines and becoming cell-phone-only households. The use of hybrid data collection methods will become a requirement for many research studies.

• Provide data without insights and marketing implication: The days of the big “data-dump” are numbered.  More and more clients are requiring insights and strategic recommendations that can be derived from the data.

• Long research reports: In the current culture of sound bites and fast Twitter-like messages, few decision makers have the will and time to sit and read thick research reports full of charts and comments. Top-line reports with only a few slides highlighting key findings and insights are becoming the norm.

• Wait for big “in-depth” research projects to gather insights: The highly competitive environment is pushing companies to increase the pace at which they make business decisions. Nobody has time to wait until large research projects are completed, and many small projects are deployed to answer tactical questions.

• Reliance on “gut feeling”: More and more companies are realizing the importance of using key performance metrics, which often include customer feedback. Often, there is too much at stake to rely only on “gut feeling” in decision making. Many companies are harnessing the power of their customer databases to gather insights on what course of action to take. Database analytics combined with primary research will be used more and more to understand customer behavior and its motivators and thus provide support for smart business decision making.

Perceptual maps are often used in brand research to represent consumers’ perceptions of brands or products on two or more dimensions represented by X- and Y-axes, each with ends that have opposite meaning (e.g. bitter vs. sweet, cheap vs. expensive).  Each brand has a position in the perceptual map space that reflects their relative similarity or preference to other brands with regards to the dimensions of the perceptual map.  With the help of perceptual maps we can transform consumer judgments of similarity or preference into distances represented in a multidimensional space. 

Perceptual maps are most appropriate if we want to:

• Explore and identified unknown dimensions affecting behavior
• Compare evaluations of brands when the basis for comparison is not known

As you set the objectives for a perceptual map, there are some practical considerations to have in mind:

• Similarity vs. Preference:  Determine whether the brands will be compared on the basis of their similarity or of their preference. Similarity-based perceptual maps reflect brand similarities and dimensions of comparison, but don’t provide any insights into why the brands are chosen. Preference-based perceptual maps, on the other hand, reflect preferred choices, but consumers may compare brands on different dimensions that don’t match the similarity-based perceptual maps. Two brands can be perceived as different in a similarity-based perceptual map, but similar in a preference-based perceptual map (e.g. two brands of hair products perceived as different, but equally preferred).

• Relevancy:  All relevant brands in a product category should be included as the omission of key brands or inclusion of inappropriate brands will have an impact on the derived dimensions and the relative positioning of the brands in the perceptual space.

• Comparability:  Make sure that the brands or products are comparable in the eye of the consumer. Brands should have a common characteristic (objective or perceived) that consumers can use for comparison (e.g. clothing brands, snacks, alcoholic beverages, etc.).

• Number of brands evaluated: It is important to strike a balance between the required number of brands to find a stable multidimensional map, an acceptable level of model fit and the effort required on the part of the respondent to evaluate a number of brands.  It is recommended to have more than four times as many brands as dimensions to obtain a stable solution and avoid overfitting problems, so trade-offs are necessary regarding the number of desired dimensions and the burden put on respondents so that data quality doesn’t degrade too quickly as it happens with long surveys.

• Approach: Perceptual mapping includes a wide range of techniques, which can be grouped in two categories:
  • Attribute-Free: In this approach, typically associated with Multidimensional Scaling (MDS), respondents provide an overall evaluation of each brand, which is used to derive spatial positions in a multidimensional space that reflect these perceptions.
  • Attribute-Based:  This approach usually employs several multivariate techniques (e.g. Correspondence Analysis, Factor Analysis, Discriminant Analysis, etc.) that require respondents to provide evaluations of specific attributes, which are used to derive an overall evaluation of the brands and spatial dimensions along which brands are positioned.

        Each of the approaches you can use for perceptual mapping has advantages and disadvantages:

• Aggregate vs. Dissagregate Analysis: Perceptual maps can be generated for individual respondents or for combinations of respondents using some process of aggregate analysis.  If we are interested in understanding the overall perception of brands and the dimensions on which they are evaluated, an aggregate analysis is the most appropriate. If, on the other hand, we want to understand individual variations, the disaggregate analysis provides better insights.

On the whole, perceptual maps are useful tools to infer underlying dimensions from similarity or preference evaluations provided by respondents about brands. They can help us determine:

• How brands are perceived in relation to each other, which is important in competitive market analysis
• What dimensions to evaluate brands are used, which allows to identify messages that resonate with the target audience
• How many dimensions may be at work in specific evaluation situations, which helps to focus brand positioning and helps identify potential marketing tactics to reach our target audience

Make sure you use a perceptual mapping approach that provides marketing managers with insights that can be put into action.