Maxdiff is a superior technique for the research of preferences or importance. In our presentation at the 2010 AMA Market Research conference in Atlanta, my colleague Kathryn Korostoff from Research Rockstar and I made the case for MaxDiff and discussed its advantages over rating, ranking and constant sum questions.

Rating questions are susceptible to:

• User scale bias: this includes acquiescence bias (tendency to agree with everything), extreme responding (using certain parts of the scale) and social desirability bias
• Scale meaning bias: a scale point can mean different things for different respondents
• Lack of discrimination: respondents often rate everything as preferred or important

Ranking questions’ limitations include:

• Order bias: we get different results depending on whether the respondent ranks the items from highest to lowest or vice versa
• It is a difficult task as respondents have to evaluate all items at the same time to determine their ranking
• Only a limited number of items can be tested without increasing the level of effort required from the respondent
• Provide ordinal data which limits the types of analysis we can do with the data
• Don’t allow for ties, which can occur in reality

Constant sum questions’ weaknesses include:

• It is a difficult task as respondents have to evaluate all items at the same time to determine the number points that they need to allocate to each item
• Like with ranking questions, only a limited number of items can be tested without increasing the level of effort required from the respondent
• Respondents engage in response strategies trying to make the task easier (e.g. allocating equal amount of points to each item; given all points to one item, etc.)

Given the problems with each of these question types, particularly with rating questions, has led to an increased interest in the use  of Maximum Difference Scaling or MaxDiff as is commonly called.

Maxdiff is a trade-off analysis technique that allows us to do multiple pairwise comparisons in an effective way by asking respondents to select the most and the least preferred or important items out of a list we want to test in search for the greatest differences among items.

MAXDIFF BENEFITS

• Strong discrimination power
• It is a simple task for the respondent
• Allows to test a larger number of items
• Eliminates scaling bias
• Allows for diversity, which is necessary in international studies
• Provides ratio data and a measure of magnitude

THE PROCESS

In order to implement MaxDiff we need to:

• Identify the number of items to test.
• Create an experimental design that provides frequency balance (each items appears the same number of times), orthogonality (each item is paired with other items the same number of times), position balance (each items appears the same number of times in each position).
• Estimate utilities for each items using Hierarchical Bayes analysis or MNL and rescale them for interpretation.

The standard output of MaxDiff analysis is usually a ranking of the items tested based on rescaled utilities, but these can also be used to conduct further multivariate analysis such as correlations analysis, multiple regression, t-testing, TURF analysis, cluster analysis, latent class segmentation, etc.

MAXDIFF APPLICATIONS

MaxDiff can be used to study preferences for and importance of a number of things including:

MaxDiff is not perfect. It usually takes longer for the respondent to take, and depending on your research goals, the relative measure it provides may not be what you want. MaxDiff helps you prioritize within a given list of items, but it doesn’t tell you if all are preferred/important or not from an absolute perspective. However, the latter is less of a problem as we can include additional questions, which allow us to calibrate the MaxDiff ranking to “absolute” levels of importance or preference.

Nonetheless, next time you need to measure preferences or importance consider using Maxdiff instead of traditional approaches such as rating, ranking or constant sum questions. You will gain in data quality, greater discrimination and the ability to provide better insights to support business decisions.

Stay tune for the upcoming case study of how MaxDiff can be used in market segmentation.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

Someone recently sent me a post from Joshua Black where he categorically advises not to ask your customers directly what they want or what the problem is, because they don’t know. Among the things he recommends for listening to customers are going to Walmart for eavesdropping customers around and walking your dog to let your subconscious mind gives you the answers.

I hope that no serious business owner pays attention to this type of advice. Try going to the bank or to a VC to get money for your business based on that. I can tell that customers CAN tell you what they want; you just have to know how to ask and what to ask.

It has become fashionable to be “anti-survey” and regard customers like lost souls living without knowing why they do anything. In a WSJ article written by Dr. Barry Berman, from Hofstra University’s Zarb School of Business in Hempstead, N.Y, he exorts businesses to not bother asking customers because you would just get an emotional response. According to Mr. Berman, you should only rely on behavioral and transactional data, ignoring that emotional reactions play a big role in many purchase decisions. Joining the club is Phillip Graves, who after a couple of years as a market researcher got dissapointed and is now on a mission to make surveys history.

In the era of cheap online survey tools when everybody thinks they can write surveys and ask questions, many companies end up with bad and misleading research results, which in turn give business owners reasons to support the position that there is no point in asking customers directly.

Another argument against asking customers is that several big brands have been born because their creators had an idea and went for it, without doing any research. Ford and Walmart are often cited as examples. Unfortunately, there are countless ideas, even patented ones, which have failed miserably because their creators didn’t ask their potential customers if they really needed the product.

Particularly now, with the fierce competition we see in most product categories, if you don’t do the right type of market research (which is more than popularity polls, focus groups and sales data) you are doomed to fail. Even if the initial idea was successful, companies like Ford and Walmart, most big brands have market research teams dedicated to listen to their customers with the right methods. Starbucks is also an example of a company with an unbelievable growth based on the CEO’s instinct. However, the Great Recession has taught him that instinct is not enough. He was notorious for despising research, but now has recognized the need of it. To read it in his own words, check his interview with Business Week in 2009: Starbucks: Howard Schultz vs. Howard Schultz.

Many small business fail because their owners had an idea and never bothered to check whether there was a real need in the market (beyond asking their closest friends, relatives and anecdotal data) that would sustain their idea. Even behavioral testing and sales data will give you only half of the story. If you really want to grow your business, you need to know not only what customers do, but also why they do it.

The aforementioned post and many who commented on it advise you to walk in the customers’ shoes and ask yourself what he would want and do. Unfortunately in this case, human perception is very selective and we tend to “see and hear” what already confirms our beliefs, so doing anecdotal research as suggested by eavesdropping or making calls to your subconscious mind, will only give you biased information aligned with what you want to hear. You are too close to your idea to be objective.

My advice: Get a market research vendor, a neutral party with experience that can first guide you to the best data collection method for your research objectives and target population. If you want to observe customers in Walmart, get researcher with experience in ethnographic research who knows how to capture observational data and can give you an objective interpretation of what you are seeing or hearing.

In short, a professional researcher will help you craft the write questions to ask, select the best data collection method (qualitative or quantitative) and extract insights from the data without contaminating them with your expectations and desires.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

This month I will be presenting at the 2010 AMA Market Research Conference in Atlanta, Sept 26 -29, with my colleague Kathryn Korostoff from Research Rockstar. Our session is about Maximum Difference Scaling (MaxDiff) and its use in market segmentation.

We will talk about what MaxDiff is, how it compares to traditional methods such as rating and ranking questions, and in what types of research can be used. We will also discuss case studies where MaxDiff has been applied in market segmentation.

Why should you care? Simply because MaxDiff helps to make better business decisions and this is an opportunity to learn why and how. Based on our own experience after conducting MaxDiff studies for many of our clients, I know that MaxDiff can deliver powerful insights and our clients feel more confident in their decision making.

To learn more about MaxDiff and its applications check Making The Case For MaxDiff .

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.