I can’t emphasize enough the need for realism in price research in order to get valid results. We are well aware that in the context of a survey, respondents are likely to over- or underestimate their willingness to pay for products and services (more pronounced in certain categories than in others), so we need to strive for research designs that minimize the gap between the survey context and actual purchase behavior.

An interesting study comparing different approaches to price research, with realism in mind, was recently published in the 2011 February issue of the Journal of Marketing Research (How Should Consumers’ Willingness To Pay Be Measured? An Empirical Comparison of State-of-the-Art Approaches by Miller, Hofstetter, Khromer and Zhang).

In the article the authors present a comparative study of direct and indirect methods to measure willingness to pay to actual purchase behavior of a cleaning product. The methods examined include:

1. OE: Direct questions asking consumers to state their willingness to pay using an open-ended question format (Direct)
2. CBC: Choice-Based Conjoint Analysis, which calculates willingness to pay based on consumers’ choices among several product alternatives and a “none” choice option (Indirect)
3. BDM: The Becker, DeGroot and Marschak approach, in which the participants are obligated to purchase a product if the price drawn from a lottery is less than or equal to his or her stated willingness to pay in response to a direct question (Direct)
4. ICBC: Incentive-Aligned Choice-Based Conjoint analysis, in which participants are also obligated to purchase a product if the price drawn from a lottery is less than or equal to the  inferred willingness to pay based on his or her choices among several product alternatives and a “none” choice option (Indirect)
5. REAL: in which participants were asked whether they would buy a product using a dichotomous format (Yes/No) at a certain price displayed in an online shop similar to that of the product manufacturer. This data was treated as real purchase behavior. The prices shown in the online shop corresponded to the price levels used in the conjoint analysis groups (CBC and ICBC).

All participants who were obligated to buy in the BDM and ICBC groups, or agreed to buy in the REAL group, were sent the product and an invoice by mail, which they could pay in cash or by credit card.

The most significant finding of this exercise is that in terms of the absolute distances between an elicited willingness to pay and the actual purchase behavior, the BDM group showed the least deviation from the REAL group, followed by ICBC, OE and CBC. In other words, the BDM and the ICBC approaches elicited a more realistic response to different price points and yield results closer to those of the REAL group, which is not surprising as these approaches put participants in an actual purchase situation.

 However, I was surprised to learn that OE did better than CBC. These results stay in contrast with other studies (Ding, Grewal, Liechty, 2005) and the results from a second dataset the same authors analyzed (digital compact camera) where CBC performed better than OE. The authors hypothesize that CBC is likely to perform better when a product is less unique and faces more competing products, which was not the case of the cleaning product tested. This makes sense and tells us that we should consider the product category when deciding which willingness to pay measurement approach we should use.

 As an overall conclusion, the authors suggest that market researchers should consider incentive-aligned approaches such as BDM or ICBC when:

• A product prototype exists
• The underlying product is not expensive
• The cost of fulfilling the actual buying obligation is low

 Although, the study itself has several limitations (assumptions of market monopoly, small number of CBC tasks, limited to one product category, low completion rates and relatively low understanding of the task under incentive-aligned conditions), it gives support to the idea that we are likely to get more realistic results in price research if we provide test conditions that are as close as possible to the actual purchase situation.

This is part of a series of pricing research posts that Jeffrey Henning and I are writing. The series so far:

- Comparing Willingness To Pay Measurements To Real Purchase

- Conjoint Analysis And Realism In Price Research

- Price Research Review

- Getting The Price Right Takes More Than Guesswork

- Making The Case Against The Van Westerndorp Price Sensitivity Meter

- The Van Westendorp Price Sensitivity Meter

- Monadic Price Testing: “Shh, It’s All About Price

- Estimating Willingness to Pay

To learn more about our price research service visit Price Research.

There are many things to consider if we want to write surveys that gather high quality data, including data collection method, respondent effort requested, question wording, order, format, structure, visual layout behaviors to be measured, accuracy of the elicited information, among others. Although all these issues are important, at the end of the day, what we want is to create surveys that yield results that are valid and reliable.

Validity and reliability are often discussed in the field of psychometrics, but not so much in market research, although it is assumed they are present. 

Validity is concerned with the accuracy of our measurement, and it is often discussed in the context of sample representativeness. However, validity is also affected by survey design since it also depends on asking questions that measure what we are supposed to be measuring.

Most surveys often have what is called face validity, which is a matter of appearances. The questions seem like a reasonable way to obtain the information we are looking for, but are they really? There are other types of validity survey writers should strive for:

• Content validity: This is related to our ability to create questions that reflect the issue we are researching and make sure that key related subjects are not excluded. For example, if we are interested in learning how consumers use hair styling products, and only ask how they have used them in the past week, we are likely to miss information about how these products are used under different weather conditions (given that humidity can give you a bad hair day in a blink of an eye) and end up with an incomplete picture of consumers’ behavior in this category.

• Internal validity: This asks whether the questions we pose can really explain the outcome we want to research. In our hair styling product example, we need to ask questions that help us identify factors that influence the selection of hair styling products. Here we are looking for a relationship between independent variables (e.g. hair type, desired hair style etc.) and the dependent variable (e.g. likelihood to buy the hair styling products).

• External validity: This refers to the extend in which the results can be generalized to the target population the survey sample is representing. As we all know, the way we ask questions will determine the answer we get, so the questions should reflect how the target population talks and think about the issue under research, which often call for the need to conduct exploratory qualitative research. In our example, if we want to estimate the share of preference our hair styling product would get in the hair styling category, we need to include other brands that represent this category, otherwise we can’t extrapolate the results to the category as a whole.

Reliability, on the other hand, is concerned with the consistency of our measurement, that’s the degree to which the questions used in a survey elicit the same type of information each time they are used under the same conditions. This is particularly important in satisfaction and brand tracking studies, as changes in question wording and structure are likely to elicit different responses.

Reliability is also related to internal consistency, which refers to the degree different questions or statements measure the same characteristic. A practical application of this concept can be found in marketing segmentation studies that try to capture psychographics and construct behavioral or satisfaction segments by asking respondent to rate a list of statements using different rating scales (e.g. agreement/disagreement; likes/dislikes; excellent/poor, etc.). In our example, if we want to identify “lovers of styling products,” the statements used to describe such consumers should provide a consistent description of this group. This can be tested by using correlations, split sample comparisons or methods such as Cronbach’s Alpha.

Validity and reliability are not always aligned. Reliability is needed, but not sufficient to establish validity. We can get high reliability and low validity. This would happen when the wrong questions are asked over and over again, consistently yielding bad information. Also, if the results show large variation, they may be valid, but not reliable. So, don’t forget to think about reliability and validity when writing your next survey and strive for reliable and valid results.

The first question I always get from clients interested in conducting a survey is about sample size. Many confuse sample size with representativeness. They are related, but not the same, particularly if convenience samples are used.

In random samples, as we increase sample size the chance each member of the target population has of being selected increases and consequently more segments of the population are likely to be represented. This is based on the assumption that we have a list with all the population members (population frame) and know their probability of being chosen. This could be the case of a customer database/list, if that’s our population of interest. 

In convenience samples, the population frame becomes the pool of individuals in the sample source (e.g. online panels), which may not include all segments in the target population or only have a few members of certain segments, depending on how the sample source is built. In this case sample quotas, weighting schemes, and mixed mode data collection methods (online/phone/intercepts) are often used in an effort to reach representativeness. 

Assuming that we are able to pull a representative sample of the target population by whatever affordable means are available to us, we need to give serious consideration to sample size. This is a case where size matters (pun intended). Why?

It is all about precision, tolerance for risk and cost. For samples smaller than 1000, we always have to think about how confident we want to be that estimates are within a particular range (level of confidence and risk), and how small we want that range to be (level of precision). Unfortunately, they go in opposite directions. Higher levels of confidence require greater ranges (margins of error) in small sample sizes.

For instance, we can be 95% confident that the true estimate for a variable in a sample of 400 is within +/-4.9%, however, if we want a smaller margin of error, in an attempt to gain more precision, with the same sample size, we have to sacrifice certainty and may need to accept a 90% confidence level to get a +/-4.1% margin of error. At the 95% confidence level you are more certain, but less precise as you expand the range to make sure the true value falls in it. At the 90%, you are more precise, but less certain.

If you want to get more precise estimates without sacrificing certainty in the results, then you have to increase sample size, which in turn increases research costs. As the table below shows, as sample increases the differences in margin of error across the different confidence intervals become smaller.

At the end of the day, when it comes to sample size, you need to decide what it is more important to you, certainty or precision, and what your tolerance for risk is, especially if your market research budget is small.

For more help on calculating sample size and margin of error, use our Sample Size and Margin of Error Calculators.

Written in collaboration with Kathryn Korostoft.
Published in the American Marketing Association Newsletter, 2/3/2011

In a fall 2010 study conducted by the authors on behalf of the American Marketing Association, in preparation for the 2010 Market Research Conference (AMA MRC), we surveyed 882 AMA members, about market research topics they were interested in and challenges they are facing in their market research roles.

In this article we present the results from client-side researchers, and the implications for market research agencies. By examining the results from the client-side respondents (416 of the 882 participants), the opportunity exists for market research agencies to get a sanity check on how well they are doing in terms of addressing their clients’ current priorities.

Top Challenges

The top three challenges identified by the client-side subgroup were:

• Limited market research budget (61% respondents)
• Lack of time and staff to thoroughly analyze research results (51% respondents)
• Lack of time to learn new market research methods (44% respondents)

These results represent an age-old issue of not enough time and not enough money, which is not surprising. If we contrast them with the lesser rated items, we begin to see that those results are more than just superficial. Some of the lesser-important items reported were:

• Finding and selecting new market research reporting tools (24% respondents)
• Finding vendors that stay on top of trends in our industry (19% respondents)
• Finding skilled market researchers to staff internal market research team (10% respondents)

Keep in mind these results were in the form where people could select all that applied. When forced to select only one of the items, limited market research budget stays at the top at 24%.

Top Topics of Interest

Market researchers were also asked about what topics would be of interest to them for future conferences or learning events. Using the Maximum Difference Scale (MaxDiff) approach, we tested 26 market research topics and ranked them by preference. These results show that client-side researchers were mostly interested in:

• Market segmentation
• Branding research
• Latest trends in the market research industry
• Social media monitoring and sentiment analysis
• New online qualitative market research techniques

Items of less interest? Pricing research, mobile research, and ethnography (at 23^rd!). Of course, these are also items that are more relevant to some industries than others, which likely explain their overall lower scores. So just as agencies promote segmentation to clients, they also need to do it themselves.

Bottom Line

Many market research agencies invest in producing webinars, white papers, and other content in order to educate clients and establish brand credibility during the sales cycle.  Using these research results, they can align the topics with actual client priorities.

For more information, please contact the researchers directly:

• Relevant Insights – Michaela Mora
• Research Rockstar - Kathryn Korostoff

Not long ago I received a survey from a trade association testing the appeal of market research online training courses using concept testing . They also wanted to know how much I would pay for these courses using the Van Westendorp Price Sensitivity Meter (PSM ).

In my opinion, the best approach they should have adopted in this case is Conjoint Analysis, which provides useful insights for product development, competitive positioning, brand equity measurement, market segmentation and specially price research.

Conjoint analysis includes a set of trade-off analysis methods such as Full Profile Conjoint Analysis, Adaptive Conjoint Analysis, Choice-Based Conjoint Analysis (CBC) and some other variations of the latter (Partial Profile CBC, Adaptive CBC). Which method should we use? The one that better reflects how buyers make decisions in the marketplace . (Getting started with Conjoint Analysis, Bryan Orme, 2006)

Choice-Based Conjoint Analysis (CBC) is widely used since it tries to mimic the actual purchase decision process for products within a competitive context. In CBC we present different product configurations to respondents and ask which they would choose or purchase. Products are configured using a set of relevant attributes, including price.

As an example, to conduct a Choice-Based Conjoint Analysis for the aforementioned online market research training classes, we would follow the following steps.

• Identify key attributes (or factors) and levels. This is probably the most important step in the design of conjoint analysis studies since the selection of the proper attributes will have an impact on our ability to reflect how buyers make purchase decisions. Qualitative research or other research sources should be used to provide realistic attributes. The presentation of attribute levels is not restricted to text. We can use images when appropriate. According to Orme, the attributes or factors should:
  • Cover the full range of possibilities for existing products
  • Be independent from each other  with no overlapping meaning
  • Be mutually exclusive
  • Have a balanced number of levels across attributes (when possible)

The table below has an example of attributes that could have been included to test online market research courses. 

• Create an experimental design. This should provide frequency balance (each attribute appears the same number of times), orthogonality (each item is paired with other items the same number of times), and position balance (each items appears the same number of times in each position). The experimental design is then used to generate a certain number of choice tasks for respondents. Below is an example of a potential choice task for online market research training courses.

 Using this approach for the case of online market research courses would not only provide a more realistic scenario for respondents, but also prevents them from focusing solely on price, decreasing the natural tendency to lowball when price becomes the center of attention, as it happens in price research approaches using direct questions about willingness to pay, purchase intent or the Van Westendorp PSM.

•  Estimate utility coefficients. These are measures of desirability for each of the attribute levels and can be estimated with different methods including aggregated multinomial logit, latent class analysis, or Hierarchical Bayes estimation (more commonly used these days).

• Develop a market simulator. This uses the utility coefficients to predict which product configurations are more likely to be chosen among many product configurations, including those that were not presented to respondents. This is the most valuable tool for managers as they can conduct “what-if” analysis to predict shares of preferences for different product configurations.  

In the case of the online market research course, the market simulator could indicate that Option 1 in the example below gets the highest share of preference likely due not only to a lower price point, but also to the availability of a Q&A option. Option 3, which doesn’t have a Q&A option, with a similar price point to Option 1, gets the lowest share of preference. Even Option 2, which offers Q&A with a live instructor, receives a higher share of preference despite having the highest price.

Simpler price research approaches are appealing because of their simplicity, affordability and ease of implementation, which suit many managers that don’t have much experience with statistics and may not feel comfortable with advanced techniques such as Conjoint Analysis. Unfortunately, simpler approaches often don’t reflect how people make buying decisions and can result in misleading conclusions.

As for cost, the field of conjoint analysis is constantly evolving and new techniques have become more accessible and affordable to research practitioners as computational tools become commercially available, so cost should not be a big barrier as has been in the past to the use Conjoint Analysis for more realistic price research.

This is part of a series of pricing research posts that Jeffrey Henning and I are writing. The series so far:

- Price Research Review

- Getting The Price Right Takes More Than Guesswork

- Conjoint Analysis And Realism In Price Research

- Making The Case Against The Van Westerndorp Price Sensitivity Meter

- The Van Westendorp Price Sensitivity Meter

- Monadic Price Testing: “Shh, It’s All About Price

- Estimating Willingness to Pay

To learn more about our price research service visit Price Research.

As published in the Market Research Association’s Alert! Magazine, November 2010.

There is an intense discussion among marketing researchers about whether DIY research is going to replace external marketing research vendors. Sometimes it feels that way. The rapid proliferation of DIY marketing research is making some external marketing researchers anxious about their livelihood, while others are simply angry about the trend towards “good enough” research which can poten­tially produce biased data and misguided conclusions.

 Reduced marketing budgets, access to relatively cheap online survey tools and the precarious position that market­ing research holds in many companies have led many to see DIY research as an acceptable alternative and sometimes the answer to management’s prayers to get at least some data on the cheap.

 I know how it is. I was the ultimate DIY research department at several of my former employers on the client side. I was able to provide high quality primary mar­keting research at a very low cost, and a quick turnaround. I estimate I saved those companies millions of dollars in research. And that’s what companies are expecting, particularly in difficult economic times. Notice that I say “high quality,” “low cost,” and “quick turnaround.” How? Simple: Expertise and experience.

 Did I replace research vendors? Yes and no. My team could tackle many research projects that could have otherwise gone to external marketing research vendors. But there were other instances where I needed to call vendors.

After being on both sides of the river, I think external marketing research vendors will still be needed for years to come despite the spread of DIY marketing research.

 Hiring marketing research vendors can be beneficial because they can:

• Offer expertise and skills not avail­able within the DIY market research team. To be totally self-sufficient a DIY marketing research team would need to have a staff of researchers with expertise and hands-on experience in a wide range of research methodologies, which is costly and nearly impossible. They also need researchers that can interpret results in meaningful ways to offer actionable insights. This requires industry and marketing expertise. These are expensive too. Those who say they do “everything” internally are often limited to do certain types of research with moderate to low level of complexity. You recognize them easily, when the answer is “We have SurveyMonkey.” Sooner or later they go looking for a research vendor that can help them extract deeper insights from collected data or simply provide a different approach that better answers questions related to the business issues at hand. When the stakes are high, management is aware that “good enough” research is simply not enough.

•  Provide validated methods and innovative approaches. Many large research agencies have established processes and standardized methods that have been tested over time, which provides consistency in data collec­tion and reporting (e.g. brand tracking research). There are also boutique research vendors that can provide more custom approaches and innovative solutions (often accompanied with a more personal­ized attention and commitment). DIY research teams can add value to the service they offer internally by selecting external vendors that have spent the time and resources on developing processes, learning new methodologies and gather­ing experiences from different industries that can guarantee data quality and meaningful insights.

• Alleviate workload for time-strapped DIY marketing research teams. In my experience, there are no small marketing research projects. Even in the simplest projects, there is a lot of work to do. Nowadays, DIY marketing research teams are often small and don’t have time to handle all the internal requests. They may need to outsource parts of some projects or entire projects as other priori­ties take place. We see this every day. For some clients we help only with research and questionnaire design and they do the rest. Others ask us to design and program conjoint studies, while they do the simula­tions themselves. Sometimes we come at the end of the process to analyze the data or write the reports, while in many occasions we take on projects from start to finish.

• Provide neutrality. External market­ing researchers are neutral judges of research results and don’t have a vested interest in its outcome one way or the other. We don’t feel the pressure to get some hypotheses confirmed and our compensation doesn’t depend on it. Our unique position as a neutral party helps to prevent unbiased results. DIY research teams can also play a neutral role, but sometimes they can’t help but infuse certain bias without knowing it based on industry knowledge and office politics. I have seen many instances in which a DIY research team’s internal clients, hoping to get a particular outcome, explicitly request the use of a particular methodology, (for some reason focus groups are always top of mind), which may not be the right one to tackle the business issue in question. Sometimes the DIY team is able to push back, but at other times they can’t and the internal client wins.

• Guarantee the anonymity of respondents and confidentiality when this is an issue. Think of employee research. This often touches on sensitive topics regarding satisfaction with management, salary, co-workers, etc. Using a third-party to gather and analyze data helps to protect employees’ identity and avoid any type of retaliation against them. Sometimes this also applies to customer satisfaction research. There are cases in which customers might be unwilling to air certain opinions about a company if they know they are being surveyed by the company’s employees and can be easily identified. Even if customers know the research is being sponsored by a company, knowing that a third party is collecting the data and analyzing it mitigates confidentiality concerns.

• Provide credibility to research results. Public data gathered internally is often received with some skepticism, particularly if it is favorable to the company’s brands and products, so companies are better off by using a neutral third-party vendor that can assure the data is legit. This can also be needed when internal teams in an organization ask for resources to launch a product, invest in an idea, etc. They need a credible data source to explain the rational of their request, and at times this means data gathered and interpreted by a knowledgeable neutral third-party.

•  Offer access to target populations of interest. There are more and more companies who are mining their cus­tomer base and using them as the primary source of sampling. Some companies are creating online communities they can re­cruit from for research purposes. However, often companies must go outside their customer base in order to identify growth opportunities. External research vendors can then find populations out of their reach. Even companies who build com­munities need help to create and manage them.

 There is the belief that DIY marketing research is always cheaper than hiring an external vendor. If we look only at the cash that could have been paid to an external vendor, DIY research is definitely cheaper. However, I would argue that DIY research is not always cheaper. I have seen many cases in which the research needs to be done again due to bad research designs at the hands of inexperienced DIY research teams. In times when many think they can write a survey because of access to cheap survey tools, the need for expertise is often disregarded costing thousands of dollars in wasted resources on decisions made on less than reliable data. In estimating the actual cost of DIY research, companies need to factor in the alternative cost of getting bad data quality, an inadequate research approach and misleading conclusions.

 In short, consider hiring an external marketing research vendor when:

• The DIY marketing research team doesn’t have qualified staff to design and execute the type of research needed or the expertise to interpret the results and extract actionable insights.
• A different research approach is needed to get deeper insights into the issues at hand.
• The DIY marketing research team doesn’t have time to handle all research requests.
• A neutral third-party is needed to avoid biased results.
• Anonymity and confidentiality are a concern.
• A credible party may be needed to legitimize the results.
• The DIY team doesn’t have access to the population of interest.

The increase of cell phone-only households and use of cell phones most of the time, even by those who still have landlines, is becoming a serious issue market researchers involved in survey research can’t overlook.

Why can’t we ignore cell phones? It is a matter of coverage, as Scott Keeler, the Director of Survey Research at the Pew Research Center  pointed out in a recent presentation to members for the Market Research Association (MRA). He called attention to the fact that at least 25% of adults can be reached only on a cell phone, proportion that is even higher is certain groups such as young adults 25 – 29 (51%), 18 – 24 (40%) and Hispanics (34%). This means that when we rely only on landline sample frames, certain population segments are likely to be underrepresented. Analysis of data from the Census and Pew Research Center surveys shows that only 7% of adults 18 -29 can be found in landline samples, while 66% are adults over 50.

Including cell phones in telephone samples is more complex than just calling to landlines. Keeler pointed out a set of practical, legal and ethical issues we face when dealing with cell phones samples.

LEGAL

• Because of the Telephone Consumer Protection Act (TCPA) established in 1991, landline and cell phones are separated, so researchers wanting to include cell phones in telephone samples have to work with a dual frame approach.
• The TCPA also prohibits automatic dialing to cell phones unless there is prior consent from respondents, which means cell phone numbers need to be hand-dialed, making it very inefficient and costly. Landline numbers that have been transferred to cell phones need also to be treated as cell phones, which requires some phone number scrubbing.
• Callbacks to increase response rates are also limited. There is high likelihood that the same person answers a cell phone, so too many callbacks can be perceived as harassment.

ETHICAL

• Many teenagers and children have cell phones and they are more willing to answer a call, so there is a higher risk of dealing with minors. This tends to result in more ineligibles (and larger lists of cell phone numbers need to be dialed).
• Often people move and keep their cell-phone’s area code so the geographic information associated with cell phones may be wrong which complicates the time of day when a call may be appropriate.  This can results in that someone with a West coast area code living in the East coast, may get a call for a phone survey a little too late for his or her like.
• We also need to consider respondents’ safety at the time we call them since they may be driving or doing something in which if they get distracted it may endanger them.
• Respondents’ privacy is another issue to take into account. We may be asking sensitive questions respondents don’t feel comfortable answering if they are in a public area or close to people with whom they don’t want to share certain information.

To deal with these ethical issues, telephone surveys with cell phone samples need to be designed to carefully screen respondents  and verify to whom we are talking, if it is the right time to call, if it safe for the respondent to talk and if the person is in position to talk about sensitive topics.

DATA QUALITY

Concerns about the quality of the data for cell phone samples have also been voiced. Audio quality during the interview, potential distractions for the respondent, respondents’ multitasking, possible desire to rush to complete the interview, and sensitive topics are among the factors that could have a negative impact on data quality in cell phone samples.

HOW MUCH DOES IT COST?

Unfortunately, including cell phones in telephone samples doesn’t come cheap due to the inefficiencies brought by the above mentioned legal issues and the pricing structure of cell phone services in the US, where consumers pay not only for the calls they make, but also for the ones they receive.

According to the American Association for Public Opinion Research (AAPOR)‘s Cell Phone Task Force, cell phone interviews are typically at least twice as expensive as landline interviews, and can be three or four times more costly if cell phone-only samples are used.

To find guidance in how to meet the difficult and costly challenges of cell phone surveys check the 2010 AAPOR Cell Phone Task Force Report: New Considerations for Survey Researchers When Planning and conducting RDD Telephone Surveys in the U.S. With Respondents Reached via Cell Phone Numbers

WHAT ARE THE ALTERNATIVES? 

• Online panels: These are often used in the market research industry due their ability to provide affordable samples, albeit most offer non-probability samples. They may provide coverage of some of the cell-only population, but there are concerns about data quality and representativeness.
• Address Based Sampling (ABS): These are household samples drawn from USPS Delivery Sequence File (the “DSF”) which can be used for telephone, Internet, mail or in-person interviewing. ABS covers 98% of the households in the US and 60%-70% can be matched to a phone number. There seems to be a growing use of this approach by big polling entities and survey cost could be comparable to RDD samples, but more expensive than online panels.

 Finally, if your organization can only afford landline samples, luckily, according to Keeler:

•  Most analyses have found no differences in data quality between cell phone and landline samples.
• There is some bias in coverage in landline-only samples when compared to samples that include cell phones, but the bias is not too big. In an analysis done by the Pew Research Center for several of their surveys, they found, as the chart below shows, that most of the time the difference between both sample frames was 1 or 2 percentage points. Its relevance depends on what we are measuring. If we are talking about polls predicting an election, a small bias can be of great significance, but it is possible that for many commercial research studies, it doesn’t have the same impact.
• Response rates in landline samples are similar to that of cell samples, and both have similar level of respondent cooperation.

 Although these are good news, they don’t provide a solution to the coverage issue, so if you are interested in reaching young groups that can’t be reached by landlines, and can’t afford a cell phone sample, you should consider a hybrid approach that combines telephone and online surveys or give ABS a try.