as published on November 18, 2011 by the Dallas Business Journal

For entrepreneurs, it is never too early to start thinking about which marketing metrics to track when running a business.

Some of the most basic but important marketing metrics can be gathered through survey-based Awareness, Attitudes and Usage research.

This type of research provides quantitative marketing metrics about product and service knowledge, perceptions and customer behavior. The research is often set up as a “tracker” used to monitor brand health and trends affecting business growth, but it can be implemented as an ad-hoc research project to gather feedback from the market at any time. Typical question areas include:

• Awareness: Both top-of-mind (“Which brand comes to mind when you think of …?”) and recognition (“Have you heard of Brand X?”). Awareness metrics are important to decide where and how to invest advertising dollars.

• Attitudes: Beliefs and feelings about the brand, company or products/services. We need to understand how our brand is perceived to direct our marketing communication to either strengthen or change those perceptions.

• Usage: What, when, where, how much, how often do you buy/use Brand X and Y? Knowledge about the purchase behavior of customers and noncustomers allows us to quickly become aware of and react to any changes in the market as new marketing strategies, products, and prices are put in place not only by us, but also by old and new competitors entering the category.

As these marketing metrics are gathered, entrepreneurs should also put systems in place to capture transactional data linked to customer profiles (such as demographics) so results can be validated through triangulation with actual sales data. While sales data can tell us what customers do, it doesn’t tell us why they do it. The metrics can fill some of the gaps inherent to transactional data and support better business decisions.

Whether you are starting a business that needs a website to represent it or already have a website that needs a makeover, you should consider testing different design alternatives before launching into development. Don’t assume you know your audience if you have never tested how it would interact with your website. Testing website designs doesn’t have to be complicated or very expensive.

In the concept stage you can do it as simply as using still pictures of different designs showing how it may look like. Do it with real content and as close as possible to a finished look. Even if the interactive piece is missing in this approach, you will learn tons about the first impression a design gives and its capacity to grab attention and invite visitors to come in and explore your website site.

By comparing website designs in a simple concept testing you will learn how different elements speak to your audience, including:

• Color scheme
• Layout
• Content organization
• Navigation expectations
• Potential sources of abandonment
• Brand image elicited by the design
• User identification as a member of your target audience (Is this website for me?)
• User concerns and anxieties about using your website
• User reaction to different calls to actions and labels

Testing these elements with your target audience (there is no such thing as ‘my product/service is for everybody’) early in the design phase will save you time and money in the long run as you will be able to launch a website that will work for your business from the get go. Of course, testing shouldn’t stop there as website interaction is a big factor in the user experience, and for that you may need qualitative and qualitative usability testing. However, if your budget is tight, the more reason to start with a simple design concept test. It would be better than just relying on personal taste or misleading assumptions about your target market.

To learn more how we can help you with this type of research check our Website Design Optimization Service. You can also check the case study about what we did we our website.

I hear questions related to statistical significance on a daily basis. It is usually some variation of “How much sample do we need to be significant?” which often reflects some confusion about the term.

Statistical significance is a concern when we are interested in detecting differences not due to chance between two or more groups (people, objects, ads etc.) being compared.

As sample size increases, the margin of error around a percent or a mean get smaller and we get, not only more precise estimates, but also more sensitivity to detect differences that are not due to chance. In a large sample, a difference of 1 or 2 percentage points may be significant, while in a smaller sample, where there is more variation, we may need to see more than 10 percentage points to detect significant differences.

In survey research, we often talk as if the results are finite point estimates when in fact we should be talking in ranges since there is always a margin of error around any estimate. So if the margin of error is +/-3% and we get a value of 50% for a variable, it means that the true value of the variable should be between 47% and 53%.

Now, if we measure the same variable in another group with a sample size where the margin of error is +/-5% and we get a value of 57% for the same variable, it means that the true value is expected to be between 52% and 62%. Despite the 7 percentage point differences, which seems large, we can’t say that it is statistically significant because there is some overlap between the margin of error range of each group (47% – 53% and 52%- 62%) and the true value of the variable in the second group could be 52% or 53% which are values included in the first group’s margin of error range.

How confident are we about this? We often say 95% confident, which means that if we repeat the study 100 times, we can expect similar results 95 times and be wrong 5 times. This is called Confidence level and the margin of error range is called Confidence Interval. In short, we want to make sure the true value falls within the same range every time we repeat the study. Unfortunately, statistical confidence has an inverse relation with estimate precision. If you want to be 99% certain then you have to allow for a larger confidence interval that will include the true value.

If there is no comparative analysis involved, it doesn’t make any sense to talk in terms of statistical significance. However, we are still concerned about estimate precision of results in total. We want our margin of error to be as small as our budget and tolerance for risk allow. To get greater precision, we need larger sample, which in turn costs more money. To be more certain, you sacrifice some precision. There is always a trade-off to make.

Next time when you considering sample size for a survey get ready to answer these questions:

• What is the desired precision (margin of error)?
• How confident do you want to be?
• Can your budget accommodate the required sample size for the desired precision? If not, what are you willing to settle for?
• Are doing any comparisons between groups? If so, how many?
• Can your budget accommodate the required sample size by group to make meaningful comparisons?

Unfortunately, the difference between the sample you want and the one you can afford is oftent significant (pun intended), so budget questions are always in the mix. For more help on calculating sample size and margin of error, use our Sample Size and Margin of Error Calculators.

At the root of survey gamification are good, sound survey design principles. That’s the main message from Reg Baker’s presentation at the MR Festival.

Baker follows the cognitive process (Tourangeau, Rips and Rasinski, 2000) involved in how respondents process information and survey questions and points out the opportunities to create engaging surveys. When faced with survey questions respondents go through different phases:

• Comprehension: Understand the information, apply logic, connect key terms. Survey design can help comprehension by keeping questions simple, avoiding vague concepts, being specific, defining ambiguos terms and providing examples.

• Retrieval: Memories are retrieved and blanks are filled in. Survey design can make retrieval easier by offering cues, providing keys to important events, and decomposing the situation the question refers to.

• Judgement: Asses relevance, integrate material, draw inference. Survey design can aid judgement by managing the context to which the question applies, decomposing the question and discouraging overthinking questions.

• Response: Categorize, edit responses. Question formatting can improve response rate by avoiding certain question types (e.g.  open-ended, numeric questions, long grids) and using meaninful scale anchors.

Survey tool providers are racing to create different question formats (e.g. sliders, heatmaps, etc.) to make the survey-taking experience more engaging and minimize abandonment rates. However, with the increase of surveys and DIY research done by inexperienced people, the quality of survey design has declined. Writing surveys looks easy, but it is not. Fun and cool question formats can’t compensate for ill-designed questions.

I have to agree with Baker that the greatest improvement needed now to engage respondents is in survey design.