The concept of mixed-mode surveys is nothing new, but it seems to be gaining traction in the research community. Among the issues pressing the use of mixed-mode survey designs are the need to reduce coverage bias, increase response rates and lower costs.

REDUCE COVERAGE BIAS

The advent of new technologies combined with fast-paced demographic and cultural changes have put a dent in the ability of traditional and newer data collection methods to cover all segments of the population.

• In-person interviews. The increase of gated communities and locked apartment buildings in tandem with safety concerns has made it very difficult to access people’s homes for in-person surveys.

• Telephone surveys. The trend toward not listing addresses in phone directories; increased use of answering machines with Caller ID; and listing of phone numbers in the Do Not Call Registry have introduced problems with coverage in phone surveys. While survey research is exempted from the Do Not Call Registry, most consumers are unaware of this. Telemarketers have been blamed for much of this change. In the ’90s telemarketers began using some of the techniques used to conduct surveys, making people less willing to answer unsolicited sales calls and let their phone numbers be public.

• Population coverage. Population coverage via phone surveys has also been affected by the proliferation of multiple phone lines in homes (i.e., fax, Internet), cell phones and voice-over-IP lines. Landline-based samples can’t guarantee high coverage rates any longer. Research from the National Center for Health Statistics indicates that by the end of 2009, at least 25 percent of adults could only be reached on a cell phone – a proportion that is even higher in certain groups (i.e., young adults ages 18-29 and Hispanics). Analysis of data from Pew Research Center’s dual-frame surveys also shows that only 7 percent of adults 18-29 can be found in landline samples, while 65 percent are adults ages 50 and older.

  

  

  

• Online surveys. Web surveys have grown exponentially in the last 10 years. At the same time, online surveys have always raised concerns about coverage. Although a large portion of the population has Internet access, it doesn’t mean they are all accessible for online surveys. Most primary market research conducted online uses samples sourced either from online panel providers or from clients’ own customer databases.

  Online panel samples are, by definition, convenience samples. Even when their composition may resemble the general population, they are populated with people who want to participate in surveys in exchange for incentives. We can select random samples from the panel population but the fact remains that not all individuals in the population will have the same chance to participate in surveys, as not all belong to an online panel.

  There are always groups that will be underrepresented (e.g., Hispanics, young males, low socioeconomic strata, etc.) or overrepresented (e.g., white, educated, etc.) in the online panel population. A typical example is Hispanics, separated by different acculturation levels. Most online panels have acculturated, bilingual Hispanics but very few, if any, unacculturated Hispanics. Consequently, if we want to survey less-acculturated Hispanics we have to use phone or in-person interviews to reach them.

Mixed-modes can be used to reach respondents and invite them to take the survey in another mode. This is the case of postal mail, in-person and phone interviews used to recruit respondents for online surveys, which can be taken at a central location or at home, if the person has Internet access. By combining different data collection modes – and reaching different segments with the mode that is more effective for each – we can improve coverage.

INCREASE RESPONSE RATES

• Data collection fit. Survey response rates across all data collection modes have experienced a steady decline for decades. Often a mixed-mode approach is used to increase response rates by providing different alternatives to participate in surveys, depending on respondents’ preferences or resources. Some people may not have access to a computer at home or may not feel skilled with computers; in these instances a phone or in-person interviews are more appropriate than online surveys. In studies about sensitive topics with people that can only be reached via in-person interviews, computer-assisted personal interviews can be mixed with computer-assisted self-administered surveys so that the person can answer – directly on the computer – questions that feel too embarrassing to answer out loud in front of the interviewer.

• Survey length. Another factor, which usually affects response rates negatively, is long surveys, which unfortunately have become too common. Faced with tight budgets and short deadlines, many research clients often cram as many questions as possible, particularly in online surveys, trying to capture data on all possible issues (current and old) in the few studies they can afford. Research has shown that as the survey length increases, response rates increase and data quality deteriorates. The best way to reduce drop rates related to survey length is to make surveys shorter, although it has been argued that good survey design can also reduce drop rates in long surveys.

  In cases when inevitably we have to ask a large number of questions, a mixed-mode approach may help to increase response rates. Examples of this are studies that require respondents to record activities or media consumption, like the National Consumer Survey conducted by Experian Simmons, in which in-person, phone and mail are used to recruit, remind and capture data.
  In our effort to increase response rates by using mixed-mode surveys, we should never lose sight of the goal of achieving a representative sample of the target market. It may be easier to increase response rates in certain segments using a data collection mode but we may end up skewing the sample.

LOWER COST

Cost is often one of factors that weighs heavily on the decision about what data collection mode to use. Multiple data collection methods can help mitigate the cost of certain research projects, especially if the study requires reaching low-incidence groups or groups not accessible with a certain mode.

Currently, online surveys are the most cost-effective of all data collection methods, so they are often combined with other more expensive modes like phone or in-person interviews to reach smaller subsets of the sample to increase coverage. For instance, in tracking studies that require samples from the general population and the Hispanic segment, we may direct the general population sample and part of the Hispanic sample to online surveys, while the rest of the Hispanic sample is recruited by phone-to-Web, phone-to-central location or phone-to-central location-to-Web. These combinations cost less than if we were to conduct only phone or in-person interviews with the whole sample, Hispanics and non-Hispanics. Of course, none of these combinations would cost less than doing online surveys only but sometimes this is not feasible without introducing coverage bias and non-response errors.

INTRODUCES MEASUREMENT ERROR

Despite all the benefits of mixed-mode surveys, researchers and clients must be aware of a major limitation of using multiple data collection modes in the same study. The use of different modes introduces measurement error because people answer questions in different ways depending on the collection mode. There are three key factors that usually influence how people answer.

Question formulation
Question wording and format tend to be different between data collection modes. For instance, in selection questions, while respondents are able to read answer options in mail and online surveys, the same questions asked over the phone often require changes in wording so they sound natural when spoken.

In mail and online surveys, respondents tend to put more thought in the items listed early but as they go down the list and more information is added it becomes more difficult to consider all the options at the same time (Krosnick and Alwin, 1987). In this case, respondents tend to choose among the first items listed if they are in agreement with what the respondents had in mind (primacy effect). In phone interviews, on the other hand, respondents don’t have enough time to process all the items being read and the last answer options heard are more likely to be remembered and selected if they confirm what the respondents are thinking about the subject in question (recency effect).

To avoid burdening respondents’ memories with too much information, multiple-choice questions are usually asked as a series of forced yes/no questions in phone interviews. However, the same questions are commonly asked as multiple selections (i.e., “check all that apply”) in online surveys. It is common practice to treat the results from both types of questions as equivalent but in fact, they are not. Research has shown that respondents answering forced yes/no questions spend more time processing the question and mark “yes” more than with the check-all question format when used in phone and online surveys (Smyth et al., 2003). This suggests thatforced yes/no and check-all formats are not comparable.

Sense activation
Respondents’ comprehension of a question is greatly influenced by the senses activated by the data collection mode. In phone surveys, respondents have to actively listen and remember the questions and answer options. As they interact with the interviewer, they can’t avoid being influenced by the meaning conveyed by the interviewer’s tone of voice, intonation, accent and other characteristics. In online surveys, visual representation of words and graphic elements such as page layout, colors, font size and images become more important and have a greater impact on how respondents answer. In short, there is always a potential risk that the same question asked in an online survey and in a phone interview will be interpreted in different ways by the same respondent yielding different answers unless efforts are made to make sure the same meaning is conveyed in both modes.

Rating questions seem to be particularly sensitive to oral and visual presentations. Many studies (Dillman et al., 1984; Krysan et al., 1994; Christian, 2007) have shown significant differences in responses to rating questions when asked by phone, IVR, online surveys and self-administrated paper surveys. The results in all the studies indicate that the same rating questions may produce more positive ratings when the respondent is not able to see the rating scale, regardless of scale length; use of labels on all rating points or just the end points; or respondents rating in one or two steps (Dillman et al., 2009).

Presence of interviewer
Data collection modes such as phone and in-person interviews can’t escape the influence of the interviewer on respondents’ answers, as the activation of social norms is inevitable during the interaction. When interviewers are present, respondents are more likely to acquiesce and provide socially-desirable answers than when interviewers are absent – even if not very sensitive questions are asked.
Another factor to consider is the fact that interviewers control the delivery of the survey. They ask the questions and can probe further to better understand respondents’ answers, which is not possible in paper or online surveys. Further probing can improve measurement but it creates more interaction with the interviewer and potentially a higher risk for bias driven by social norms. Acquiescence and social desirability can also be present in online surveys depending on how the questions are formulated but the risk is mitigated by the absence of interviewers.

PART OF THE FUTURE

Whenever it makes sense and is feasible, we should always try to keep surveys within a data collection mode to minimize measurement error produced by the use of multiple methods but mixed-mode surveys are part of market research’s future.

The solution is to find a balance between the need to make the questions work for each specific mode and the need to create questions for each mode that produce comparable results. This was precisely the goal of the survey design guidelines developed for achieving commonality across mail, Web, phone and handheld computer modes in the 2010 U.S. Decennial Census. The underlying principle for these guidelines was described as:

“Universal Presentation: All respondents should be presented with the same question and response categories, regardless of mode. While one might assume that this principle requires questions, categories, instructions, etc., to be identical across modes, this assumption turns out to be neither feasible nor desirable. Rote repetition would result in awkward and difficult-to-administer instruments that are unlikely to achieve consistent response data. Rather, Universal Presentation says that the meaning and intent of the question and response options must be consistent. In some cases, questions or instructions need to be modified so they can be communicated to, attended to and understood by respondents the same way in different modes. The goal is that instruments collect equivalent information regardless of mode. By equivalent, we mean that the same respondent would give the same substantive answer to a question regardless of the mode of administration.”

WORK ACROSS MUTIPLE MODES

Many times the differences between how questions are designed are a matter of tradition and researcher specialization in one data collection mode. Market researchers need to think about designing surveys that work across multiple modes and provide comparable results.

 Multiple-choice questions (check all that apply) are one of the most common question formats found in online surveys. However, there are a couple of problems with this type of question:

• It often makes it easy for respondents to engage in satisficing behavior, which occurs when respondents select the answer options without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement, rather than work on finding the optimal answers that best represent their opinion.

• We really don’t know what it means when an item from the list is not chosen. This could happen (Sudman and Bradburn, 1982) because:

1. The option didn’t apply to the respondent
2. The respondent is neutral or undecided
3. The respondent overlooked the item

 WHAT CAN WE DO ABOUT IT?

A solution to this problem is to ask multiple-choice questions as a series of forced yes/no answers for each of the question items. This format requires that respondents report a judgment on each of the items. Research has shown that forced yes/no questions encourage deeper processing time and discourage satisficing response strategies as measured by the time spent on answering forced yes/no vs. check-all questions and the number of items marked affirmatively in each question format. Research by Smyth et al. (2003), comparing results from both types of formats in online surveys has found that:

• Respondents who answered forced yes/no questions spent significantly more time responding than did respondents to the check-all formatted questions.

• The forced yes/no format yielded more options marked affirmatively than the check-all format.

We can argue that the longer time spent answering the forced yes/no questions is a mechanical function of the fact that respondents are forced to give an answer for each item and spend extra time marking “no,” which is not required in the check-all question.

However, the positive correlation between time spent on answering the question and the number of options selected has also been shown to be an indicator of deeper processing and more thoughtful answers for the check-all formatted questions as well. Respondents who spend more time answering check-all questions mark significantly more answers than those who answer check-all questions in less time.

Another research result supporting the hypothesis of deep processing is that no significant differences have been found in the number of options marked affirmatively between respondents that take longer time answering yes/no questions and check-all questions.

The yes/no format for multiple-choice question are not a 100% foolproof, as some respondent may still show satisficing behavior by marking yes or no for all options or marking them randomly. In this case we need to put quality checks in place during programming that take into account the time spent on the question and any straightlining patterns.

An issue that we also need to manage is the fact that sometimes respondents are undecided or think an option doesn’t apply to them. In this case it would be wrong to force them to give a yes or no answer. The best remedies against this problem are respondent screening and survey skips that would avoid showing options that don’t apply. In cases in which there is still room for this problem, I recommend adding a third “Don’t Know/Not Applicable” option.

IMPLICATIONS FOR MIXED DATA COLLECTION MODES

 The forced yes/no format for multiple-choice questions is commonly used in phone surveys since it is impractical to read all the options to respondents and expect them to remember them all to answer the question. Often, when mixed data collection modes are used, (phone/online, phone/paper), the yes/no and check-all question formats are treated as equivalent, assuming they are answered the same way. Research suggests that this would be a mistake.

Experiments carried out by Smyth et al. (2008) with phone and online survey using both question formats have shown that the forced yes/no format yields consistently more options marked affirmatively than check-all formatted questions in online self-administrated and phone-administrated surveys. This supports the idea that results from both question formats are not comparable and shouldn’t be treated interchangeably.

KEY TAKEAWAYS

• You are better off using forced yes/no format for multiple-choice questions in order to elicit deeper processing and minimize satisficing behaviors.

• Do not mix the yes/no and check-all formats across data collection modes, as results are not comparable.

The advent of user-friendly online survey tools in recent years has created the illusion that anybody can write a survey questionnaire. After all, how hard can it be? It’s like asking questions in a conversation, many think. However, there are many methodological issues to consider when creating a questionnaire if you want to gather high-quality data in a survey. The following are 10 issues that arise in survey design.

• DATA COLLECTION METHOD

Some questions may elicit different answers if asked in an online survey, a telephone interview, a paper survey or a face-to-face interview. While words in phone surveys or in-person interviews are given more importance because of the conversational format, visual design elements have a bigger impact in how questions are read and interpreted in online surveys. Be aware of the types of questions that are a good fit for online surveys.

• RESPONDENT EFFORT
  

There are questions that put a heavier burden on the respondent’s working memory and comprehension or are likely to elicit higher non-response if asked in different data collection modes. Experience tells us that asking a ranking question with 10 items over the phone can overwhelm respondents. In online surveys, rating questions in matrix format with a large number of items increases fatigue and boredom and often leads respondents to adopt a “satisficing” behavior. Satisficing occurs when respondents select the same scale-point to rate all items without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement, rather than work on finding the optimal answers that best represent their opinion.

• QUESTION WORDING

Formulating a question with the right wording so it accurately reflects the issue of interest is one of the hardest parts in writing questionnaires. You may have seen political polls getting different answers depending on how a question is crafted. Data errors can creep into a survey if we use unfamiliar, complex or technically-inaccurate words; ask more than one question at a time; use incomplete sentences; use abstract or vague concepts; make the questions too wordy; or ask questions without a clear task.

Another issue related to question wording is the risk of introducing bias by leading the respondent in a particular direction. I recently received a mail survey sponsored by the Republican Party to represent the opinion of voters in my congressional district and one of the questions was:

“Do you think the record trillion-dollar federal deficit the Democrats are creating with their out-of-control spending is going to have disastrous consequences for our nation?”

Could this question be more biased? The use of adjectives such as “record,” “out-of-control” and “disastrous” makes it really clear what the expected answer is and what the intentions of the sponsor are.

• QUESTION SEQUENCE

Questions should follow a logical flow. Order inconsistencies can confuse respondents and bias the results. For instance if you are measuring brand awareness and ask respondents to recognize brands they are familiar with before asking which brands first come to mind, you are rendering the results from the latter question worthless since respondents can’t avoid thinking of brands they just saw in the first question. This seems basic, but it happens.

• QUESTION FORMAT

Questions can be closed-ended or open-ended. Closed-ended questions provide answer choices, while open-ended questions ask respondents to answer in their own words. Each type of question serves different research objectives and has its own limitations. The key issues here are related to the level of detail and information richness we need, our previous knowledge about the topic, and whether to influence respondents’ answers. For example, for closed-ended questions we need to decide what the answer choices should be and in which order they should appear. This requires we know enough about the topic to provide answer options that capture the information accurately.

• INFORMATION ACCURACY

Questions can be closed-ended or open-ended. Closed-ended questions provide answer choices, while open-ended questions ask respondents to answer in their own words. Each type of question serves different research objectives and has its own limitations. The key issues here are related to the level of detail and information richness we need; our previous knowledge about the topic; and whether to influence respondents’ answers. For example, for closed-ended questions we need to decide what the answer choices should be and in which order they should appear. This requires we know enough about the topic to provide answer options that capture the information accurately.

Some questions yield more accurate information than others. Respondents can answer questions about their gender and age accurately, but when it comes to attitudes and opinions on a particular issue, many may not have a clear answer. Overall, attitudes and opinion questions should be worded in a way that best reflects how respondents think and talk about a particular issue so that we can tease out information that is difficult for the respondent to articulate. However, some questions need to be skipped when they don’t apply to the respondents’ experience or the issue is so irrelevant to the respondent that s/he doesn’t have a formed opinion about it. In the case in which attitude statements appear grouped in a matrix format and some may not apply to a respondents (e.g., a customer satisfaction survey after a phone call to customer support), it is necessary to include a “Not sure/Don’t know/Not applicable” option to avoid introducing measurement error in the data.

For intance, the other day I received an online customer satisfaction survey from BlackBerry after a call I made to its support desk. The survey had a question in which I was asked to rate the representative who took my call on different aspects. One of them was “Timely Updates: Regular status updates were provided regarding your service request.” I wouldn’t know how to answer this, since the issue I called for didn’t require regular updates. Luckily, they had a “Not applicable” option, otherwise I would have been forced to lie, and one side of the scale would be as good as the other.

• MEASURED BEHAVIORS

People tend to have less-precise memories of mundane behaviors they engage in on regular basis, and usually they do not categorize events by periods of times (e.g., week, month and year). We need to consider appropriate reference periods for the type of behavior we want to measure. Asking “Have you purchased any piece of clothing in the last seven days?” will yield a more accurate behavior measure than asking “Have you purchased any piece of clothing in the last six months?”

Measured behavior should be relevant to the respondent and capture his or her potential state of mind. This is valid particularly when we use rating questions and have to decide whether to include a neutral mid-point. A lot of research has been conducted in this realm, particularly by psychologists concerned with scale development, but no definitive answer has been found and the debate continues. Some studies find support for excluding it while others for including it depending on the subject, audience and type of question.

Those against a neutral point argue that by including it we give respondents an easy way to avoid taking a position on a particular issue. There is also the argument that equates including a neutral point to wasting research dollars, since this information would not be of much value or at worst it would distort the results. This camp advocates for avoiding the use of a neutral point and forcing respondents to tell us on which side of the issue they are.

However, consumers make decisions all day long and many times find themselves idling in neutral. A neutral point can reflect any of these scenarios: we feel ambivalent about the issue and could go either way; we don’t have an opinion about the issue due to lack of knowledge or experience; we never developed an opinion about the issue because we find it irrelevant; we don’t want to give our real opinion if it is not considered socially desirable; or we don’t remember a particular experience related to the issue that is being rated.

By forcing respondents to take a stand when they don’t have a formed opinion about something, we introduce measurement error in the data since we are not capturing a plausible psychological scenario in which respondents may find themselves. This is yet another reason to include a “Not sure/Don’t know/Not applicable” option in addition to a neutral point.

• QUESTION STRUCTURE

  Questions have different parts that must work in harmony to capture high-quality data. These are the question stem (e.g. what is your age?), additional instructions (e.g. select one answer) and response options, if any (e.g. Under 18, 19 to 24, 25 +). The wrong combination can leave respondents baffled about how to answer a question. Consider the example below:

  Overlapping answer options

  What is your household income? Select one answer.

  1. Under $25,000
  2. $25,000 to $50,000
  3. $50,000 to $75,000
  4. $75,000 +

  So, which answer should I choose if I my household income is $50,000? Is it option 2 or option 3?

  Conflict in meaning between different parts of the question

  Please indicate the products you use most often. Select all that apply.

  1. Cell phone
  2. Toaster
  3. Microwave oven
  4. Vacuum cleaner

• VISUAL LAYOUT

Using design elements in an inconsistent way can increase the burden put on the respondent in trying to understand the meaning of what is asked. For example, encountering different font sizes and colors across questions forces the respondent to relearn their meaning every time they are used.

Also, presenting scales with different directions (positive to negative or vice versa) in rating questions within the same survey increases measurement error as respondents often assume all rating questions have the same scale direction even when the instructions explain the meaning of the end points of the scale. For instance, if a preference question using a 1-7 scale where 1 means “the most preferred” is followed by an importance question, also using a 1-7 scale, but where 1 means “the least important,” respondents who are not paying attention to the instructions (which is quite common) are likely to assume that the 1 in the importance question means “the most important.” I have seen many examples of this problem, when respondents are asked a follow-up question conditioned on their previous answers and then they realize their mistake and tell us they actually meant to say the opposite.

• ANALYTICAL PLAN

Based on the research object, both the type of information requested and the question format are important for the type of analysis we plan to perform once the data is collected. If you want to develop a customer satisfaction model using linear regression analysis and the dependent variable is an open-ended question, you can forget about modeling anything. This seems obvious, but I have seen non-researchers writing questionnaires without thinking how they will analyze the data and then come to me asking for analyses that are not appropriate for the data collected.

There is also the question of whether we want to replicate the results, track certain events or just run a one-time ad hoc analysis. If the goal is to track certain metrics, time and care should be dedicated to crafting tracking questions, as slight changes in wording can change the meaning of a question and thus its results.

ON YOUR WAY

If you take each of these aspects of survey writing into consideration, you will be on your way to creating surveys that produce valid data and can support with confidence strategic and tactical decisions for your business.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.