Having a website for your business is a must these days. There are many options to create one. You can do it yourself or hire someone who can do it for you. Whoever does it, your website needs to:

TALK TO YOUR AUDIENCE

A website should be designed with an audience in mind. It’s probably not for your friends and family, so you need to gather basic insights into your target market, including:

• Demographics (B2C) / Firmographics (B2B)
• Shopping behavior (what, where, how often they buy)
• How they use products/services like yours
• Competitive alternatives used
• Unmet needs
• Willingness to pay for product/services like yours

CREATE A GOOD USER EXPERIENCE

With so many options available, user experience has become an important factor in attracting and retaining customers, no matter if your website is for e-commerce or just to inform about your product and services.

There are many factors that go into creating a good user experience, but here are some basic ones you should test, at least with concept testing if not with usability testing:

• Navigation
• Content organization
• Page objectives
• Eye path
• Colors
• Readability
• Multimedia use
• Search capability

PROVIDE RELEVANT CONTENT

Content is king these days. People are constantly looking for answers to questions and you want to be there to provide them. Content is more than filling space in your website.To be ahead of the competition, create trust and be relevant you need to understand what these questions are by studying your target market. With the help of analytic tools (e.g. Google Analytics, Woopra, Omniture, Webtrends, etc.) you can track popular search terms in your product category, but you will learn tons by listening to your audience using qualitative and quantitative research.

BE UNIQUE

Being unique is not about having a flashy website, but providing a different user experience while meeting your audience’s needs. Note that uniqueness can work against you, if you try to break the rules in a way that confuses and frustrate your audience. You can create uniqueness through content, layout, and graphics, but never forget your audience. How can you know what works? Test, test, test.

RESIST THE PRESSURE

Entrepreneurs are often pressed to put up a website quickly and cheaply. Small budgets and the pressure to go to market as soon as possible often drive them to ignore some these issues. However, overlooking them will have a negative impact on your business and cost you more in the long run in terms of lost revenues and additional costs to redesign the website.

IN SHORT

Invest in testing upfront during the design phase before wasting time and money on developing a website that won’t work for your business.

To see an example of the test we did for our own website check: Web-Site Optimization Research

I recently got a request for advice via Twitter with this question: What % of segment needs to be interviewed to gain reliable insight for product optimization?

Reliability has to do with consistency of results across data collection instruments and points in time when the data is collected. I see this question being more about validity and representativeness which is related to population heterogeneity and sample source.

To determine the sample size of a segment we need to ask:

• How homogeneous is the so-called segment? Are there any sub-segments that need to be represented? Usually, the larger the segments the more heterogeneous they tend to be. As heterogeneity increases, the need for a larger sample increases as well, so all subgroups are represented.

• What is the sample source? Representativeness has to do more with where the sample comes from than with sample size. If you get it from the appropriate source with the right screening criteria you are a step closer to more valid results, although there are other factors that affect validity.

• Is the segment going to be compared to another segment(s)? We should avoid too small samples if we are going to make comparisons since smaller samples have larger margin of errors. This means that the range, in which the true value of a parameter is found for a segment, is large and may overlap with the range for the true value in the segment we are comparing it to. The end points of each range are what we call Margin of Error. If we compare two small samples and can’t detect any significant difference it may be due to overlapping margins of error, not to actual lack of differences.

• What is the level of risk we are willing to take? As we increase sample size the margins of error get tighter and precision improves. But how confident do we want to be in that the true value is indeed within the margins of error? Here we need to consider the Confidence Interval (C.I). The most commonly use is 95%. This simple says that if we repeat the study 100 times, in 95 times we should get similar results and we can expect to be wrong in 5 of 100.

• How much certain and precise do we need to be? The thing is that Confidence and Precision go in opposite directions. If we want to increase our certainty that the true value falls within a range of values, we have to widen the range (margins of error), but this leads to a lost in precision.

Depending on budget and timeline constraints you could use two approaches to sampling for segments:

1. Create quotas by segment. These act as independent groups, like smaller “total samples.” These quotas can be proportional to their size in the population or could be all the same size. In the latter case, you would need to weight the segments if you decide to merge the quotas in a total sample, otherwise some segments will be overrepresented and others underrepresented.




2. Let the segments fall naturally in the total sample. This approach can be more expensive since you will need a larger total sample if you need large enough samples by segment to be able to do comparative analyses. If no comparisons will be carried out, then this is a more desirable approach to get all segments represented in the average values.

As you can see, estimating the sample size for a segment is not different from estimating the size for the total sample and there is no magical % to determine how large the sample size should be. Sorry.

Sometimes I’m asked to review surveys or analyze data collected via surveys developed by clients and more often than not I find rating scales, (aka Likert scales) of different sizes and directions within the same survey. When I ask why, I get answers such “It” or “This is the one we have always used.”

It seems rating scales are often chosen based on preference or habit (e.g. legacy surveys), which is not surprising since there is no consensus on what rating scales work best. They all yield different results, which is disheartening in a way.

There has been a lot of research dedicated to this subject illustrating there is no simple answer to the question on which rating scales we should use.

Source: International Journal of Social Research Methodology, Vol. 13, No.1 Feb. 2010, 17-27 (Hartley and Betts)

This extensive body of research shows that different rating scales are bound to yield different results as we are mainly dealing with human perception. Rating scales mean different things to different people and the values, words, and order in which we present them have an impact on how they are interpreted. What to do?

• Whenever possible, favor question formats other than rating scales. For example, Maxdiff has been shown to discriminate better in preference and important measurements.

• If you still have to use rating scales, strive for consistency and use them with full knowledge of the bias they introduces in the data, particularly if you want to analyze data from different rating scales and data from different surveys. This is particularly relevant in tracking studies. A change in rating scale from one wave to another may show artificial significant differences mainly due to the measurement error introduced by the change in scale.

• Above all, triangulate the results with other data sources to understand how different scale points correlate with actual behavior, and ask why the person gives a particular rating. If possible use a text analytics tool to get at the heart of what the scale really means for a respondent. The example below says it all.

Twitter is becoming a great educational tool. I was invited to teach a Twitterversity class under the topic “Principles of Market Research Project Management”, a Twitter-only event organized by Research Rockstar. For those who missed it and for those who attended and want to see all the tweets I sent under the hastag #MRXU on 7/28, in one place, here they are.

Consider the following 7 steps during the implementation of a market research project:

STEP 1. DEFINE THE RESEARCH OBJECTIVES

• This is the most important step. It sets the direction of the whole market research process.
• Ask clients how they will use the research results, what business decisions they will make based on the data. They should be specific
• Do some background research, interview key stakeholders and research users to put the research objectives into a greater context
• Gain consensus among key stakeholders on the main research objectives. Get them involved from the start
• Avoid objective creep. Don’t try to research everything under the sky in a project. Focus on what’s needed for decision making
• Trying to cram many things in a project because of budget constraints is often a waste of money as data quality suffers
• Discuss limitations early in the process. Set clear expectations of what the research will cover and what data it will provide
• Check if previous research has been conducted on the same issue to avoid effort duplication and waste of money
• DO NOT select data collection method before establishing clear objectives and identifying target population

STEP 2. DETERMINE THE BUDGET

• How much are the key stakeholders willing to invest in the requested research? Get a number!
• If there is no commitment to a budget, you will be wasting your time (RFP) and your research vendor’s time (proposal)
• There is always a trade-off between research quality, deadline and cost. Make your internal clients aware of that
• There is a limit to “better, faster and cheaper” in market research. Push it too hard and you will get fast, cheap, crappy research

• Who do you want to gather data from? Customers? Non-Customers? Category users?
• Sample definition helps decide on what data collection method we use. More than one method may be needed. To read more about mixed-mode  data collection check: Mixed Data Collection Modes – Round-Up
• Create clear screening criteria. Discuss them with key stakeholders. Make sure they align with the research objectives.
• Discuss the caveats and limitations of the sample definition and how they will affect the results and decision making
• Be realistic. Given your budget, you may or may not be able to reach your target population

STEP 4. DEVELOP THE ANALYTICAL PLAN

• Based on the decisions that will be made, determine what type of data is needed and expected
• Select analysis techniques that help you reach the research objectives and provide data that research users are expecting
• Example: Need to know how to price a new product before it goes to market? Conjoint analysis may be a good fit. Check: Conjoint Analysis And Realism In Price Research
• Example: Need to pick the product name that elicits the highest purchase intent from a list of 30? Consider MaxDiff. Check: Making the Case for MaxDiff
• Example: Need to find new growth opportunities? A segmentation research can help to find segments with the highest potential. Check: Segmentation is Key to Success
• Think objectives firsts, methods second. Not the other way around
• Determine sample size based on your tolerance for risk.  Check Sample Size Matter
• A large sample doesn’t guarantee representativeness. Check: Does A Large Sample Size Guarantee A Representative Sample?
• The analysis techniques selected will also influence the decision on sample size

STEP 5. DEFINE DATA COLLECTION METHOD

• Objectives, sample, analytical plan & cost have the highest influence on which methods we use
• Be open to use hybrid approaches combining qualitative and quantitative data collection methods
• Ideally, if budget permits do qualitative research before or after quantitative research
• Consider qualitative research for exploration before quantitative and deep diving after quantitative research
• Consider quantitative research if a go/not go decision will be made. DO NOT make these type of decisions based only on qualitative research
• Discuss which methods are the best fit to research the target pop. Some target groups may be difficult to reach with the same method
• If you decide on mixed-mode surveys, be aware of potential measurement errors each mode introduces. Check: Understanding the Pros and Cons of Mixed-Mode Research
• Once the data collection methods are selected, determine if you can do it with internal resources or need a research vendor
• If time, staff or lack of tools are limitations, consider outsourcing the project to an external research vendor. For more on this check: When Do You Need A Market Research Vendor?
• If you have access to a customer database with emails, use it for studies related to customer retention goals and new product development
• For customer acquisition efforts use samples of non-customers in the category
• If the study is online get bids from multiple online panels.
• Don’t buy third-party email lists and blast them with survey invites. It is illegal (SPAM-CAN Act)
• If you are doing surveys, put time into its design. To create surveys that gather quality data check: Using A Strong Questionnaire To Harvest High-Quality Data
• Considering focus groups? Check if it makes sense here: When Using Focus Groups Makes Sense
• If you are doing focus groups, avoid common mistakes. To know which they are, check: Common Mistakes When Doing Focus Groups
• Don’t forget about new online qualitative research techniques. Check: Online Qualitative Research Techniques Review

STEP 6. COLLECT DATA

• Get involved, monitor. Do a soft launch if you are doing online surveys to catch any potential problems

STEP 7. ANALYZE & REPORT

• Keep the key objectives in mind to connect market research to business impact. Check: How To Connect Market Research To Business Impact
• Share preliminary results with key stakeholders, discuss, check if they make sense from a practical stand point
• Focus on the story behind the numbers and how it supports your recommendations. Don’t do a data dump. Focus on insights

The recent Big (D)esign conference in Dallas, which gathered many in the graphic design industry, website developers, and gamers, dedicated a track to usability. This track included a presentation by Ryan Smith from Qualtrics. Smith discussed how survey tools are evolving to provide a better user experience and match what’s happening in other areas where technology is setting the pace. Using examples from Qualtrics, he demonstrated how survey tools are allowing us to be:

• More interactive: Different questions types have emerged in an attempt provide a more interactive experience to respondents, engage them, increase response rate and response accuracy. Examples include:
  • Rating questions using sliders, images instead of numbers of text and gauges, among others

  

  • Constant sum questions using draggable bars.

  

  • Text Highlight and Heat Maps.

  

• More intelligent: Technology allows us now to do on-the-fly analysis to personalize the experience and integrate survey results with other data sources on a timely manner. We can use advanced skip logic to present relevant questions and capture session information to improve data quality (e.g. removing speeders).

• Better subject selection: Online surveys can be distributed in different ways offer better access to the target audience. Surveys can be deployed using pop-up, pop-under, pop-over, feedback links, embedded links, and link redirects. This allow us to go where the respondents are.

• More immediate: Organizations can quickly conduct surveys with internal resources (DIY research). Survey results can be monitored since the moment the survey is launched. Online reporting capabilities allow us to get updated results continuously. The extent to which we limit or extend result sharing is under our control.

There is no doubt that technology has made it possible to create more engaging surveys, clean the data on-the-fly, facilitate access to surveys, and provide quick results. However, before you jump on the wagon of the cool question types, consider these issues:

• Need: Determine which interactive features are necessary, which ones are more of a novelty, which ones do not help respondents directly to complete the survey, and which ones increase respondents’ burden.

• Culture: When introducing graphic elements like images (e.g. smilies) or graphic representations of objects (thermometers, gauges, etc.), consider who your target audience is and what cultural barriers may influence reactions to these questions. This is especially true in international studies.

• Expectations: People are used to traditional question types, so it may be unclear what it is expected from them. Clear instructions about the scale meaning and actions the respondent need to take are recommended. This may not be a problem, once these question formats become the norm, but for now, we need to ensure respondents know how to interact with the questions.

• Impact on completion rates and data quality. The time needed to get familiar with the new question format, process the instructions (even if it takes a few seconds) and question complexity may affect completion rates and data quality. Research on this is a mixed bag, so to be on the safe side, test the questions with a few members of the target audience before a full launch.

• Reaching people with disabilities: The new question types are often not 508 compliant, making it difficult for people with disabilities to participate in surveys. If you need to reach respondents that may have disabilities, stick to the traditional formats.

One of the presentations I enjoyed the most at the recent 2011 Market Research Annual Conference in Washington DC was the one by Barry Blyn from ESPN. He provided superb examples of how market research should be implemented and add value to an organization.

First, Blyn made an important distinction between Measurement and Insights. Measurement tells you what people did, while Insights tells you why they did it and how to get them to act in the future.  In my view this is where market researchers can add most value.

ESPN’s research efforts are led by these guiding principles:

1. Research projects are about insights working more like weathervanes than speed cops. It is more important to know where the business is going (weathervane) than measuring how fast things went by (speed cop).
2. Research projects are led by neither quantitative nor qualitative work. They work together as peanut butter and jelly. Results from any research method are presented together to answer the business problem.

According to Blyn, ESPN has become fanatical about listening to their audience. In 3 years they have conducted more than 400 in-depth interviews all over the country and 15,000 surveys. In search for insights they have been combining traditional and innovative types of research with consumers trying to get at the heart of the business challenges ESPN faces. Blyn presented two examples of how these guidelines are implemented:

• Brand Health Tracking (Crusade): This is a brand tracking program which is seen as an orchestra where different research methods are thought of as instruments and are combined to reach the ultimate goal: monitor ESPSN’s brand strengths and hunt for weaknesses making sure ESPN is always in touch with fans’ perceptions about the ESPN brand. In this program, ESPN combines qualitative and quantitative methods such as:
  • RDD telephone interviews with 1,500 adults 12+
  • Quarterly online surveys with 1,600
  • Backyard barbecues with fathers and sons (group discussions)
  • Conflict groups with ESPN fans debating ESPN detractors
  • “College campus crawls” (Group discussions with students about sports over beers and wings at a college bar)
  • Word of mouth/social media  monitoring
  • Attending a baseball games with fans
  • Brand Eulogy (Imagine ESPN has passed away and deliver a eulogy)

• Deprivation study: This is by far the most provocative research approach Blyn presented. In this case, they paid $400 to 60 avid sport fans (30 who only used ESPN TV and 30 who use ESPN TV and other platforms to get sport news) for abstaining from watching ESPN during football season. The main objectives were to:
  • Get vivid illustrations of how consumers felt about the brand
  • Identify where fans were going when they couldn’t watch ESPN TV (identify content “blindspots”)
  • Build a five-tool player, meaning being a brand that can do it all for everybody in different platforms

In this study, ESPN allowed participants to provide feedback through different channels: video and audio journals, focus groups (“therapy” sessions), before and after surveys, and in-depth interviews. From this research it became clear that ESPN needed to align its different brand properties from a fan centric point of view.

In a time when many believe the market research industry have just missed the train, the ESPN case shows that to make research relevant to organizations today, market researchers need to:

1. Become more insight-centric. Bring the insights into the spotlight and leave the research methods used in the background
2. Provide forward-looking insights
3. Use hybrid research approaches and triangulate results

I think we can all do that.

In one of the keynote speaker sessions at the recent 2011 Market Research Association Annual Conference, in Washington DC, a delightfully loud and dynamic banter went between Marshall Toplansky from Core Strategies and Bill Neal from SDR Consulting trying to answer questions such as:

Is marketing research really able to deliver on the critical needs of today’s (and tomorrow’s) enterprise?

Most of the discussion revolved around the pros and cons of using social media in market research. Toplansky  focused on the need for a real-time flow of data, which social media can provide, while Neal called researchers to be the voice of the customer and make sure the data we gather, no matter the channel, represents the target market.

According to Toplansky, the type of information the traditional market research provides is irrelevant to decision makers. Big corporations, which are responsible for 85% of the market research expenditure, are funding real-time, continuous flow of information on which they can make decisions on (e.g. lead generation, sales, promotions, competitors’ impact, etc.), which has been made possible by technology. For Toplansky, it is about harnessing technology and providing daily information at a cost that is equivalent to years of doing tracking research.

Neal, on the other hand, argued that our role as researchers is to find out what is going on in the market place and why. According to Neal, the” why” is not being addressed by the new technology.  He is also an ardent supporter of sample representativeness. No matter how large the amount of information we may be able to collect via social media, we always have to ask if it is representative of our target market, which Neal claims, it is unlikely to be.

Our role as market researchers is, said Neal, to be the voice of the customer, but unfortunately market researchers don’t have a place in the C-suite. Most researchers work for the CMO, but the money is managed by the CFO. This is often reflected in research guided by the “I have to have it now” mentality which often leads company astray. Unfortunately, the people who are users of consumer data are not able to judge its quality, said Neal. Corporate researchers have to act as the guardians of data quality.

For Toplansky, representativeness is more about finding consumers who are engaged with a particular brand or product category than demographic representation of the population. Companies don’t care about the general population, but about the consumers who will buy their products.

Since many companies are run quarter to quarter, argued Toplansky, it is about building the business around empirically read mass observations, and correlating sales numbers, channel-through sales numbers and other relevant metrics. The reason we are not in the C-suite, said Toplansky, is because researchers don’t speak that financial language. We fail to translate consumers’ preferences and understanding into a continuous flow of data needed for decision making.

For Toplansky, traditional market research is too slow and expensive, so finance managers default to the information they have at hand under the mantra that “some data is better than no data,” and “it doesn’t have to be perfect.” This attitude is very detrimental to the perceived value of the market research function.

Although both speakers sounded like coming from opposite points of views, I found that their arguments complemented each other:

• Social media can provide invaluable consumer data, but we need to make sure that what we hear in social media represents our target market, that is the consumers we are after, which may or may not be representative of the general population.

• Social media research can be a source of valuable insights to explain the results from traditional research and even help to design better data collection tools (i.e. surveys, discussion guides, IDIs).

• Social media can help to explain the “why” behind sales and market share data. This requires better text analytics tools that allow us to dig deeper in the content provided by social media.

• Triangulation of different data sources, including social media, is needed to translate consumer research into financial terms and increase market research’s predictive power. There is not a research method that can do this on its own.

• Market researchers need to find ways to provide more timely and relevant information to support decision making.

• Market researchers would benefit from learning more about marketing and the company’s financials to be able to make market research more actionable.

In my view, the market research industry has not missed the train. The train has just arrived and we are all trying to figure out how to hop onto the wagon without leaving valuable knowledge behind at the same time as we inspect how this new train works and where it can take us.

One of the workshop sessions I attended on the first day of the 2011 Marketing Research Association (MRA) Annual Conference, in Washington DC., was about mixed data collection modes. One of the hottest topic in our industry today.

ROUND I

Annie Petit from Conversitions/Research Now showed examples of how online surveys, text messaging and social media research can be used in combination to get the best out of each. I totally agreed with Petit when she said there are not perfect methods. We have to use all methods that are available and get the best out of each.

For example, surveys are great for frequencies, and representative demographics; text messaging is good for on-spot live behavior, to check what people are seeing, eating, drinking right now; and “social mediaresearch is like doing a 10-hour survey,” which can give us tons of data, from thousands and millions of people on any one topic. We can use them all together. Examples:

• Use social media research to identify the best options to put on a grid question (e.g. list of the most popular restaurants)
• Use sharing of media research with respondents as an incentive
• Combine social media and survey with diaries via text messaging (SMS) to understand behavior
• Invite people to a SMS survey from an online survey
• Use social media research to do a deep-dive into the results from an online survey or a SMS survey

ROUND II

In the same spirit, Stephen Murrill, from Meta Research presented  a case study of the mixed mode approach used for the California State Fair, which was interested in understanding:

• Satisfaction drivers
• Barriers to attendance
• How to motivate families to maintain the tradition of coming to the Fair
• What activities may attract teenagers

The Fair usually used exit surveys to measure satisfaction with the event, but now they needed to talk to teenagers and Non-Fair-goers for whom exit surveys were not an option. The solution was to combine data collection on-site using iPads to get feedback from Fair-goers, SMS surveys to reach teenagers that came to the Fair, and phone and online surveys to reach Non-Fair-goers.

ROUND III

Sima Vasa from Paradigm Sample and Leslie Townsend, from Kinesis Survey Technologies, joined forces to present a case study where the primary data collection mode was mobile surveys. The problem was presented by a CPG client wanting to gain insights about consumers of convenience stores (C-stores), particularly Millennials (18 – 34). The goal was to create a consumer panel of C-store shoppers and keep them engage to learn more about these consumers over time.

Since this age group often doesn’t participate in online surveys, the solution was to develop a customized mobile panel application, using the Kinesis platform, that users would download to their cell phones. This application allowed to send surveys and capture the C-store experience of this age group. The application was optimized for many different devices and detected which device were used, which allowed to target the surveys appropriately, so if a longer survey was sent, they could advise users to take the survey on a PC instead of on their cell phones.

ROUND IV

Finally, Rick Kelly from Opinionology/Survey Sampling, talked about a study done for the Tour of Utah cycling race to understand the impact of advertising during the race on purchase intent. Participants were given the chance to participate in the study  using one of three modes: a mobile survey (SMS), an online surveys, or an IVR survey. The mobile survey had 12 questions, while the other two modes had 20 questions. The survey was open during a whole month, achieving 10% response rate.  Among the results presented, we learned that the SMS survey was the most common mode used buy those answering the survey at the event, which were also younger respondents. Most of those who answered the survey online did so within 24 hrs, and participants over 55 were overrepresented among IVR survey respondents. The results also showed an increased in purchase intent over the time the survey was open, for which there was no clear explanation.  Of all the four presentations, this one left me wanting for more in order to understand how the different modes actually compared.

KEY TAKEAWAYS

Although I didn’t learned anything totally new from this workshop, the speakers made a good case in favor of using mixed data collection modes. However, we should first define what we need to know and who we need to reach and then decide which data collection modes are more appropriate for the research objectives.

Petit showed good examples of how we can incorporate social media research to support more traditional research methods. Use it to refine survey questions, dig deeper about survey results, recruit respondents or provide incentives.

Murrill showed a great example of how different modes can be used to reach different target markets, while Vasa and Townsend showed how new nechnologies can help ups gain insights about hard-to-reach market segments.

What I missed was a discussion about the major issue that combining different data collection mode has, namely,  the measurement error introduced by different data collection modes as they have an impact on how people answer questions. Maybe, next year.

Attitudinal questions are common in surveys. They are often asked using an agree-disagree rating question format. The challenge is always to create statements that capture important elements of the attitudes we are trying to measure. Ideally, if the budget allows it, we should do qualitative research to gather insights into such elements and how people think and talk about them.

Even with qualitative data available, writing good attitudinal statements is not an easy task. Here are some guidelines to facilitate the process:

• Include only relevant items for the attitudes that are being measured: Adding irrelevant items or missing important ones is detrimental to the quality of the analysis.
• Write statements in present tense and avoid referencing to the past: Past tense relies on memory plus we will never know if the respondent is referring to yesterday, a week ago of ten years ago.
• Avoid giving to much information about facts or elements that can be interpreted as tales.
• Avoid ambiguity.
• Create statements that express in-favor or against opinions related to what it is being measured. Think, “I go to work every day” vs. “I love to work every day.” Do not use items that would describe different points in a continuum (“Sometimes I like to work”), as this can be confusing. If you need to do that, it means you would be better off converting the item to a separate rating question with a specific scale.
• Use direct and simple language.
•  Avoid technical jargon and use words respondents can understand.
• Make the statements short to facilitate comprehension and minimize fatigue. In long sentences, people tend to skip parts to get to the end faster, which can lead to misunderstandings. Avoid using more than one sentence.
• Use only one logical phrase per statement. Having more than one creates confusion as to what it is being evaluated.
• Avoid words like “always,” “everybody,” “nobody”. Gross generalizations are not credible. Respondents are likely to skip answering such statements or assume an artificial extreme position.
• Avoid negations and double-negations. They can lead to misinterpretations.
• Balance the number of positive and negative statements. Don’t make them all positive or negatives, which can mislead respondents in one direction or the other.

Writing short surveys is an uphill battle with many clients. Whenever the word is out that a survey will be conducted, everybody close to the subject, being the product team, senior management or operations, wants to add questions. The thought is, “since we are doing a survey let’s get as much as possible out of it.”

Unfortunately, the only thing you get out with very long surveys is bad quality data.  Why?

NON-RESPONSE & ABANDONMENT

As the survey length increases, so does the non-response bias and abandonment rate. Simply said, respondents won’t stay too long answering questions. Many won’t even start if they know the survey length (It is a best practice to announce the length of the survey in the invitation).

For those who think they can get away with it by not announcing how long the survey will be, think again. Respondents can always figure out the length from the progress bar and will drop in the middle of the survey if they perceive it as too long (even if no progress bar is shown).  High abandonment and non-response rates affect sample representativeness negatively.

In an experiment conducted by Galesic and Bosnjac (2003) to prove this point, 3,472 respondents were divided in 3 groups based on an online survey with different lengths (10, 20 and 30 minutes).  The chart above shows how the number of respondent who started and completed the survey declined as the survey length increased.

DATA QUALITY

Respondents, who are willing to endure a long survey, are at high risk of experiencing high burden and becoming “satisficers.”

Satisfacing occurs when the respondents select the answer options without giving them too much thought. They go for the most effortless mental activity trying to satisfy the question requirement, rather than work on finding the optimal answers that best represent their opinion.  Respondents may start selecting the first choice in every question, straight-lining in grid questions (selecting the same across all options) or simply selecting random choices without much consideration.  This type of behavior renders the data worthless.

The same experiment by Galesic and Bosnjac was set to test the impact of survey length on data quality, which was measured with a variety of indicators including response times, item response rate, length of answers to open-ended questions, and variability of answers to questions in grids.

Of all the indicators, item response rate (defined as the percentage completed from all questions presented in a block) was the only one that seemed unaffected by survey length, however it is unclear if the survey was programmed to force respondents to answer before going forward in the survey.  For the other indicators, the results strongly suggest that survey length affects quality.

There are powerful reasons that push clients and force research vendors to launch long surveys. Budget, time constraints, and different agendas from internal groups are some of them. However, when surveys start getting too long, clients and research vendors should take a minute to think about the implications. After all if we get bad data, we have wasted the little time and money we started with.