Comments on: Testing For Significant Differences In Convenience Samples – What Is The Point? http://relevantinsights.com/testing-for-significant-differences Grow your Business based on facts Sun, 08 Jan 2012 21:07:43 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Michaela Mora http://relevantinsights.com/testing-for-significant-differences/comment-page-1#comment-1901 Michaela Mora Wed, 12 Jan 2011 17:08:46 +0000 http://www.relevantinsights.com/?p=2608#comment-1901 Thanks for your comment Ray. I understand your point, which is fairly common practice since most client can't afford a truly random sample. The question is what constitute "too small"? For a large sample of 1000, you will find significant differences with small changes, which won't show up with smaller samples. Thanks for your comment Ray. I understand your point, which is fairly common practice since most client can’t afford a truly random sample. The question is what constitute “too small”? For a large sample of 1000, you will find significant differences with small changes, which won’t show up with smaller samples.

]]> By: Ray Poynter http://relevantinsights.com/testing-for-significant-differences/comment-page-1#comment-1900 Ray Poynter Wed, 12 Jan 2011 16:52:02 +0000 http://www.relevantinsights.com/?p=2608#comment-1900 I'd like to nuance your answer about sig testing convenience samples. A a measure of validity sig testing a convenience sample is pointless, but it can (and I think should) have a role in reliability. When we use a convenience sample the population is the collection of people who could have been sampled, for example all the members of an online panel who meet the demographic screener. Let us say we interview 1000 people from a large panel, and assume we are interested in a value that comes out at 50%. Sig testing will suggest to us that the true value for the entire, relevant, convenience sample is between 47% and 53%. (We do not know what the true population is, but we do have an estimate for the panel). If we conduct sig tests on a convenience sample we know that any changes that are too small to be significant are probably not reliable. So, we can use sig testing as a method of rejecting findings from our panel as being too small, leaving us to work with the results that are a least big enough to be reliable, and then use something else, perhaps triangulation from other sources to assess the trustworthiness of the findings. I’d like to nuance your answer about sig testing convenience samples. A a measure of validity sig testing a convenience sample is pointless, but it can (and I think should) have a role in reliability.

When we use a convenience sample the population is the collection of people who could have been sampled, for example all the members of an online panel who meet the demographic screener. Let us say we interview 1000 people from a large panel, and assume we are interested in a value that comes out at 50%. Sig testing will suggest to us that the true value for the entire, relevant, convenience sample is between 47% and 53%. (We do not know what the true population is, but we do have an estimate for the panel).

If we conduct sig tests on a convenience sample we know that any changes that are too small to be significant are probably not reliable. So, we can use sig testing as a method of rejecting findings from our panel as being too small, leaving us to work with the results that are a least big enough to be reliable, and then use something else, perhaps triangulation from other sources to assess the trustworthiness of the findings.

]]>