I recently got a request for advice via Twitter with this question: What % of segment needs to be interviewed to gain reliable insight for product optimization?

Reliability has to do with consistency of results across data collection instruments and points in time when the data is collected. I see this question being more about validity and representativeness which is related to population heterogeneity and sample source.

To determine the sample size of a segment we need to ask:

• How homogeneous is the so-called segment? Are there any sub-segments that need to be represented? Usually, the larger the segments the more heterogeneous they tend to be. As heterogeneity increases, the need for a larger sample increases as well, so all subgroups are represented.

• What is the sample source? Representativeness has to do more with where the sample comes from than with sample size. If you get it from the appropriate source with the right screening criteria you are a step closer to more valid results, although there are other factors that affect validity.

• Is the segment going to be compared to another segment(s)? We should avoid too small samples if we are going to make comparisons since smaller samples have larger margin of errors. This means that the range, in which the true value of a parameter is found for a segment, is large and may overlap with the range for the true value in the segment we are comparing it to. The end points of each range are what we call Margin of Error. If we compare two small samples and can’t detect any significant difference it may be due to overlapping margins of error, not to actual lack of differences.

• What is the level of risk we are willing to take? As we increase sample size the margins of error get tighter and precision improves. But how confident do we want to be in that the true value is indeed within the margins of error? Here we need to consider the Confidence Interval (C.I). The most commonly use is 95%. This simple says that if we repeat the study 100 times, in 95 times we should get similar results and we can expect to be wrong in 5 of 100.

• How much certain and precise do we need to be? The thing is that Confidence and Precision go in opposite directions. If we want to increase our certainty that the true value falls within a range of values, we have to widen the range (margins of error), but this leads to a lost in precision.

Depending on budget and timeline constraints you could use two approaches to sampling for segments:

1. Create quotas by segment. These act as independent groups, like smaller “total samples.” These quotas can be proportional to their size in the population or could be all the same size. In the latter case, you would need to weight the segments if you decide to merge the quotas in a total sample, otherwise some segments will be overrepresented and others underrepresented.




2. Let the segments fall naturally in the total sample. This approach can be more expensive since you will need a larger total sample if you need large enough samples by segment to be able to do comparative analyses. If no comparisons will be carried out, then this is a more desirable approach to get all segments represented in the average values.

As you can see, estimating the sample size for a segment is not different from estimating the size for the total sample and there is no magical % to determine how large the sample size should be. Sorry.

Sometimes I’m asked to review surveys or analyze data collected via surveys developed by clients and more often than not I find rating scales, (aka Likert scales) of different sizes and directions within the same survey. When I ask why, I get answers such “It” or “This is the one we have always used.”

It seems rating scales are often chosen based on preference or habit (e.g. legacy surveys), which is not surprising since there is no consensus on what rating scales work best. They all yield different results, which is disheartening in a way.

There has been a lot of research dedicated to this subject illustrating there is no simple answer to the question on which rating scales we should use.

Source: International Journal of Social Research Methodology, Vol. 13, No.1 Feb. 2010, 17-27 (Hartley and Betts)

This extensive body of research shows that different rating scales are bound to yield different results as we are mainly dealing with human perception. Rating scales mean different things to different people and the values, words, and order in which we present them have an impact on how they are interpreted. What to do?

• Whenever possible, favor question formats other than rating scales. For example, Maxdiff has been shown to discriminate better in preference and important measurements.

• If you still have to use rating scales, strive for consistency and use them with full knowledge of the bias they introduces in the data, particularly if you want to analyze data from different rating scales and data from different surveys. This is particularly relevant in tracking studies. A change in rating scale from one wave to another may show artificial significant differences mainly due to the measurement error introduced by the change in scale.

• Above all, triangulate the results with other data sources to understand how different scale points correlate with actual behavior, and ask why the person gives a particular rating. If possible use a text analytics tool to get at the heart of what the scale really means for a respondent. The example below says it all.