If you have never done formal market research, it is hard to resist the desire to get all your questions answered the first time you decide to do it. If you find yourself with a laundry list of questions, hit your mental reset button and start from scratch thinking about what business goals the research would support.

The fundamental question is: How  will you use the results from the research?

The key to answer this question is to discern between “nice-to-know” and “must-know” information. This is easier said than done, particularly if no previous research exists and every potential piece of data sounds important.

To determine what “must-know” information is in the context of your business goals, answer these two additional questions:

• What critical piece of information do you need to make a particular decision? This may be a go/no-go decision on a product launch, pricing strategy, advertising campaign, equipment investment, etc.
• How would you implement the research results? You may use them to design or optimize products, set prices, identify profitable target markets and how to reach them, determine what to say in an advertising or PR campaign, or how to respond to competitors’ actions.

These two questions are related, but not the same. For instance, a client needed to make a decision on how to position a new product the company was launching. The research plan was to test different positioning concepts, identify the one that resonated the most with the target audience, and provide insights to why. The client’s advertising agency was then responsible for implementing the results by translating the insights into effective ads and marketing collaterals.

In another case, I worked with a client who wanted to know if there were significant differences between two subgroups of the target market.  To find differences, we needed to increase the sample size, which would increase the cost of the project.  But when I asked if the product would be marketed differently to each group, the answer was “no.” It was clear that this information wasn’t critical to either the decision or the result implementation. The client dropped the question, and we gained focus, saving money in sample and scope.

In short, think hard about what “must-know” information is before doing market research, and you will get results that are relevant and help you make sound business decisions.

Font size and type are key elements in a website that not only have an impact on its readability, but also on the brand it represents. Here are some of the insights about fonts from research conducted in recent years:

1.  Avoid smaller font sizes (8 pt, 10 pt) for text that you really want to be read and support text sizing by the browsers. Research (Beymer, Russell & Orton, 2008) in this field, using eye-tracking methodology, has showed that larger fonts  such 14pt lead to shorter fixation times, but longer return sweep times (the time needed to go back to the beginning of the next line of text), because the lines get wider. This means that large fonts are easier to read, but lines should be kept short to minimize the return sweep times to shorten reading time. The likelihood of site abandonment increases if a user perceives that it’s taking him or her too long to go through the text on a website.

 

2. Consider the “personalities” of fonts when choosing which to use. Fonts are also attributed personality traits. They can set the tone of a website (Shaikh, Chaparro & Fox, 2006) and speak volumes about a brand.

 

3. Use ClearType whenever possible. ClearType is a font treatment used to make characters appear smoother on low resolution displays. Research (KLeiman, Choi & Bias, 2006) showed that ClearType can lead to faster reading.

 

4. Consider using a slightly atypical font for material you want to be remembered. Research has shown that less traditional fonts  help to retain information longer compared to more common fonts (Diemand-Yauman, Oppenheimer & Vaughan 2010).

 

5. Use numerals (1, 2, 3, etc.) instead of words (one, two, three, etc.) to represent most numbers. They make numbers more visible.

Choose your font size and type with care to improve not only usability, but also to support and communicate your brand.

The use of qualitative research to inform new product development was the core idea in a presentation by Jeffrey Henning of Affinova at the 2012 NewMR Qualitative Event.

This is not a new idea. Probably the most common methods for testing new consumer product ideas, at least in the initial phase, are qualitative (focus groups, IDIs, etc.). More often than it should, qualitative research is used as the only base to make decisions about which products to launch, but sometimes it is followed by traditional quantitative concept testing (monadic, sequential, etc.).

Henning suggests going a step further by doing product optimization, using conjoint analysis or evolutionary algorithms, where all ideas are considered in multiple combinations and a particular combination becomes the winning product. Henning cited research on research showing that on average, product optimization doubles new product success rates and generates more than two times revenue.

This indicates that product optimization is more effective and has more predictive power than the traditional qualitative-only and quantitative concept testing often used to identify winning product ideas. However, most of the time product testing never goes beyond the traditional methods or jumps into product optimization without the previous steps. I have been witness to that numerous times. Why?

Budget and time constraints are probably the most common barriers. Funds and timelines may allow doing qualitative or quantitative research, but not both. Another factor is lack of familiarity with product optimization methodologies like conjoint analysis. Many don’t see the connection between qualitative research and these methodologies, so I like how Henning made a clear case for it.

It makes perfect sense to use qualitative research to inform which product attributes and levels should be tested using techniques like conjoint analysis. It is actually the ideal way to do it to avoid including variables that are not relevant and can distort the results.

Doing qualitative research upfront should be seen as an investment to support valid results and increased predictive power of models developed during the product optimization analysis.

 In short:

• Do qualitative research to identify product ideas and attributes that matter and have predictive power
• Skip testing single product concepts and wondering if you picked the right one to test
• Conduct product optimization including all ideas found in the qualitative phase and test multiple combinations until find the winner

It will pay off big time.

After posting the comparison of three popular online survey tools’ free versions, many have requested a similar analysis of paid online tools. It seems there is a new tool coming up every day and comparing them all would be a rather difficult task. I decided to compare three online survey tools I have used most recently for different projects.

I have been using SurveyGizmo for a while now. As I said in my previous post, it is a great value given the number of features it offers at a very affordable price. The tool has been improving with time so it was a natural choice for comparison purposes.

Not long ago, I saw a demo of Beacon by Decipher and was pleasantly surprised by the advanced features if offered and the business model that supports it. Most online survey tools require either a monthly or annual subscription, but Beacon does not. You only need to pay for completed surveys. I decided to give it a shot and used it to program a pretty complex survey. I had the chance to become very familiar with the tool, and thought it was worth including it in this comparison.

Finally, since my above mentioned post about free online survey tools, I have had many requests asking me to review Qualtrics, which is also becoming quite popular. I happened to use it in two recent projects with very different requirements, which gave me the opportunity to test it in depth.

The list below includes some of the most often-used features in moderate to advanced survey programming and, as you can see below, all three online survey tools are pretty powerful and comparable in many aspects. That said, none of these online survey tool is perfect, and they all would benefit from continuous improvement in capabilities and usability. Most of the time they do the same thing in different ways that have an impact on  users’ learning curve.

General Overview
Number of surveys	Unlimited	Unlimited	Depends on license
Number of questions	Unlimited	Unlimited	Unlimited
Number of responses	Unlimited	Unlimited	Depends on license
Number of email invitations	Up to 500k	Unlimited	Depends on license
Multi-page surveys
Account users	Up to 20 users	Unlimited	Depends on license
Question types	32	22	38
Graphic survey templates	37	9	55
Real-time reporting
Data exporting
Data deletion	Never	Never	If account is closed
Supports all languages
Price	$159/mo. 10% Off with annual subscription	No subscription needed. Pay $0.50/complete	From Free trial. $5,000/year
Question Types
Textbox – Short text
Textbox – Numeric Format Validation
Textbox – Email Format Validation
Essay – Long text
Multiple text fields
Table of text fields
Checkbox
Table of checkboxes
Radio button
Table of radio buttons
Bipolar, semantic ale table
Drop-down menu
Tables of dropdown menus
Ranking – Table format
Ranking – Drag & Drop
Ranking – Top X# of items
Ranking – Stars
Constant sum
Continuous sum
Constant sum table
Descriptive Text/Image
Image choice (radio button/checkbox)
Video		Hosted for a fee
Contact information form
Date format
Heatmap
Image/Text Highlighting
Slider
Card Sorting
Stack Sorting
Question/Page timer
File upload
Captcha
Drill down
Combination of different question types in one table
Survey Formatting
Customizable font family
Customizable font size
Customizable font color
Customizable background colors
Add/Edit Options in Bulk
Change orientation of labels and options
Change position of labels
Editable Next, Submit, and Close buttons text
Editable Next, Submit, and Close buttons graphic elements (color, size, borders)
Optional question numbering
Fully Customizable XHTML/CSS templates
Can create own CSS templates
Auto-format as currency, percentage, or a whole number
Editable error messages
Change length, height, and width of textbox or table
Editable survey width
Editable survey after deployment
Min and Max options for numerical answers
Reorder Questions
Randomize the answers
Randomize columns in a table/matrix question
Randomize questions within a page
Randomize pages
Set how many checkboxes someone can check
Optional survey tool’s logo			Depends on license
Add your own logo /White label			Requires custom development for a fee
Create max word size
Separate answer titles and reporting values
Question grouping in blocks
Logic
Question-based skip logic
Page-based skip logic
Unconditional skip logic
Show and hide questions based on logic
Question trigger on the same page
Pipe answer titles or reporting values from a question into another
Repeat questions based on answers to previous questions
Repeat pages based on answers to previous questions
Add values from a URL variable as question options
Add custom script / Javascript
Quiz score with tally
Quiz score with pass/fail
Make questions required
Make the questions soft-required
Random assignment of respondents to different paths/cells/blocks
Quota-based logic
Loops
Report Creation Options
Real-time online report
Crosstabs	Counts, Percentages	Counts, Percentages, Means, Top 2 Box	Counts, Percentages
Statistical Testing		T-Test, Test of Proportions	Chi-Square
Graphing – Use a bar graph, pie chart or spreadsheet to view answers
Export data to Excel, CSV
Export data to SPSS
Browse all responses individually
Can change titles and internal report values
Clear all responses
Share results with others via a link with or without a password		Only with password by creating user accounts
Share results with all respondents by including a link on the Thank You Page
Add a sharing/bookmarking button
Export report to PDF
Export report to Word
Export report to Excel
Export report to E-mail
Export report to Power Point
Survey Deployment
Send email using survey tool
Set a cutoff date and time
Set maximum response count
Enable IP blocking
Redirect browser
Link to panel providers
Pass data through survey link
Add a unique survey identifier to each link
Embed surveys into a Webpage
Embed surveys using an iFrame
Collect data in offline mode
Collect data in Kiosk mode
Convert link into QR code
JavaScript option embed surveys using JavaScript
Question-based quotas
Close survey based on total quota
Remember respondents’ place in survey if completion interrupted
Other Survey Options
Pipe in values from answered questions into later questions’ answers
Pipe in values from answered questions into later questions’ text
Upload images to internal library
Embed images and media via external link
Prevent search engines from indexing survey
Thank You page
Add a “Close” button to Thank You page
Back Button Optional
Progress Bar Optional
Create hidden variables
Compliance with Section 508 guidelines of the Rehabilitation Act
Export survey to PDF
Export survey to Word
Import SurveyMonkey surveys
Create mobile-compatible surveys
Import data from other data sources
Polls
Poll templates	37 + Custom		Customizable CSS
Poll Types	Single choice, Image Choice, Ranking		Single choice, Multiple Choice
Poll result sharing
Vote protection	By IP address, by cookies, or none		On/Off
Poll Widget sizing
3rd Party Integration
Open API access			Available for certain accounts
Facebook integration
Twitter integration
MailChimp integration
MailChimp integration
iModerate integration
ClickTale integration
ExactTarget Surveys integration
Freshbooks Surveys integration
Jive Surveys integration
Eventbrite integration
GoDaddy integration
Twilio SMS integration
Cint Panel integration
WhatUsersDo Testing integration
Support
Support via email
Support via phone
Support – FAQ, Tutorials

I find SurveyGizmo to be the most user-friendly and you can do pretty advanced programming with it. They have been adding new question types and integration partners. However, the data export to SPSS and result reporting feature need improvement. For some reason multiple choice questions get some random variable names that don’t match the variable name specified in the survey, when they get exported to SPSS. Variable names don’t show up in the online reports either. Also, I wished they included statistical testing in their crosstabs.

Qualtrics is the second most user-friendly survey tool of the three. Some things are not obvious. I like the large number of question types. Many are variations of standard questions with horizontal and vertical layouts, with and without images. Among the question types, you will find Constant sum tables, Heatmaps and Hotspots (The latter is pretty useful in concept testing). They recently added a new reporting tool, which is still in beta. I like that you can add banner points to the crosstabs and drill down through different question layers, unfortunately they only give you counts and the reporting page freezes a lot.

Beacon is a powerful tool, but the learning curve is a bit steep. Thankfully, they have a very responsive technical support, and I was able to get answers to my long list of questions and issues, very quickly. What I liked the most about Beacon is the flexibility it offers to create very complex skip patterns and validations. I also liked its crosstabs. They include counts, percentages, means and Top 2 Box with statistical testing. I was mostly bothered by a somewhat cumbersome process for previewing and testing the survey, which I think is something basic that should be easier to do. The tool seems to have been developed by and for programmers since the environment is not very intuitive for neophytes. You can spend a lot of time trying to figure out how certain basic features work.

Regarding cost, each online survey tool uses a different model. SurveyGizmo’s subscription can be monthly or annual (the latter gives you a 10% discount). Beacon doesn’t require a subscription, but charges $0.50/complete. This may be a good deal compared to SurveyGizmo as long as your project’s sample or the total sample across different projects is 300 or less. If you go over 300 respondents, SurveyGizmo would be a more economical alternative with unlimited completed surveys.

Qualtrics is the most expensive of the three. The number of completed surveys will depend on license cost and you have to commit for a whole year, so consider your research volume before making such commitment. However, if you have need of some of their unique question types (e.g. Hotspot), it may be worth it.

As I said before, no online survey tool is perfect. All have quirks when you get into the details. The decision to use one or the other should be based on how well they meet the needs of the project you are working on and what the budget is.

If you are a startup or a small business in need of a website, you may decide to develop the website yourself or ask someone you know or a digital agency to do it for you. In either case, it is most likely that you will get involved in the creative process, since, after all, the website would be representing you and your business to the world.

Some of the factors that have an impact on how well the website will work for your purposes are:

• Visual appeal
• Font size and type
• Page layout
• Content complexity
• Role of images
• Navigation
• Forms and Tables
• Menus
• Links

In this and subsequent posts, I’ll be writing about the main takeaways from research done in recent years in each of these areas. My hope is to help you make informed decisions about your website’s design.

Let’s start with visual appeal. You only have milliseconds to grab a visitor’s attention and only one chance to create a first impression, which will affect subsequent visits by creating a positive or negative “halo effect.”  Research (Lindgaard et al., 2006) has shown that visitors can reliably decide which homepages they like or dislike within 50 milliseconds, and that their first impressions hold over time, even if they have more time to evaluate it later on. This suggests that visual appeal is established before the visitor is even aware of doing it, at a precognitive level.

Visual appeal has been shown to have an impact on:

• Purchase intent and likelihood to re-visit a site
• Trust and reliability perceptions
• Perceptions of negative website attributes, which sometimes are overlooked in the presence of strong visual appeal

Elements such as color and layout seem to be as important as text elements in creating visual appeal, but how they will impact the effectiveness of your website can only be found out via concept and usability testing. You can test a visual concept of your website using some of the online survey tools that are available for free or at very affordable prices. You can do it yourself or hire a professional researcher. It will be worth it, if you want an effective and visually appealing website.

For more on the subject check the post about the role of font size and type in website usability.

Nowadays customization has become a must and many products and services have adopted a “menu” approach similar to what restaurants offer. You can now find product categories in which:

• Different packages including a certain number of features are offered (e.g. Netflix, Software as a Service or SaaS online services)
• Products are offered in “combos” side by side à la carte items (e.g. McDonalds, customizable HP computers, cell phone plans, banking services)
• Customers have the opportunity to build your product from scratch (e.g. Genghis Grill, Build Your Own BMW).

The challenge is to find the right balance of offering to provide options for both Do-It-Your-self and Do-It-For-me customers. The good news is that we have now tools that allow us to mimic and research a “menu” offering faster and cheaper than ever. Welcome Menu-Based Conjoint Analysis.

This technique allows us studying customer choices in a context where customization is present and multiple choices can be made.

In traditional Choice-based Conjoint (CBC) analysis, customers are presented with a set of options and are asked to select one. The problem with this is that some of the features included in an option may not be relevant to the person making the choice.

Adaptive Conjoint Analysis (ACBC) is an improvement by starting with a set of exercises aimed at identifying which attributes are relevant and using them to create options from which participants have to choose one. Still, in both approaches, participants can only choose one option and in full customization scenarios these approaches fall short.

In Menu-Based Conjoint Analysis (MBC) we can create scenarios that match the actual offering, either combos + à la carte items or all à la carte items and let participants choose multiple options within the
parameters of the offering. We can then estimate the probability of choosing each “combo” or indivdual item presented in the menu at different prices.

Some of the business questions for which the MBC approach is a good fit are:

• How should the offering be configured? What items/features should be included?
• What item combinations are more likely to be purchased?
• How many item combinations should be offered to different market segments?
• Which prices by item or item combinations will increase purchase likelihood?
• What are the optimal premium and bundling prices?
• Which items are substitutes and which complement each other?
• Which items can be eliminated from the product line without having a big negative impact on sales?
• How will an offering structure change affect sales and profits?
• What items would increase the conversion rate of a promotion?

Products and Services that could benefits from Menu-Based Conjoint Analysis:

• Automotive
• Banking services
• Hotels
• Pharma
• Restaurants
• Software as a Service (SaaS)
• Subscription services
• Telecom services
• Website Development
• Any service or product offering customized solutions

If your business operates in any of these categories or in any other not mentioned here that offers any degree of customization, the Menu-Based Conjoint Analysis can help you not only to configure new offerings, but also optimize your current product and services to increase revenues and profits.

Customer satisfaction and customer retention, although correlated sometimes, don’t have a straightforward relation.

Customers may continue buying your products and services because of:

• Habit or inertia when the cost and risk of changing is perceived to be higher than the benefits and the customer experience may be “good enough.”
• Competing alternatives are as good or nonexistent.
• Price.
• Clearly superior products and services as well as excellent customer experience.

In customer satisfaction research, we typically ask about overall satisfaction with products or services. When inertia, lack of competition, price and partially good offerings are the main reasons why customers continue patronizing a product or service, an overall customer satisfaction score may be misleading.

To avoid being misguided by an overall customer satisfaction metric, you should include other metrics, such as likelihood to continue being a customer and likelihood to recommend the products and services to others. The fact is that no single customer satisfaction metric alone will be accurate enough.

Before putting a lot of weight on a single satisfaction score, making business decisions or basing employee compensation on it, design a customer satisfaction research plan with metrics that reflect the performance of your business in key areas, customer touch points and how you stand against the competition. Consider monitoring:

• Product/service design and performance
• Customer experience with your product or service
• Customer service
• Usage and satisfaction with competing alternatives
• Price sensitivity

When it comes to customer retention, adopt a holistic approach, use more than one metric and focus on key drivers.

There is a lot of debate about how to use social media content in market research and in which category it fits: Quantitative or Qualitative research?

The massive amount of content generated by social media and the proliferation of sentiment analysis tools attempting to quantify it, give the impression that it can be considered quantitative research and thus quantitative analysis methods (e.g. t-testing, regression, etc.) can be applied to it. However, the issues of data accuracy and representativeness have not been solved yet.

DATA ACCURACY

Anybody who has done coding of open-ended questions in surveys knows how difficult this is. Finding categories to classify the answers is a very subjective process. To attain some level of objectivity and find categories or codes that accurately reflect the content in these questions, we often need several iterations and different coders to achieve consensus.

In social media, as in surveys, more often than not, people write incomplete and grammatically incorrect sentences, and use irony, sarcasm or humor to convey their meaning. In social media it is even more difficult to discern what people are talking about, as there are no survey questions to filter the information by.

Unfortunately, there is not a good text analytic tool yet that can interpret language nuances as well as a human coder. Sentiment analysis tools still need to be fed with codes and definitions of positive and negative content defined by someone in order to count and classify content.

REPRESENTATIVENESS

Linked to data quality is the problem of representativeness of opinions. From customer satisfaction research, we know that people, who decide to provide feedback, are either very happy or very unhappy with the issue they give feedback about. Those who are in the middle often don’t bother to comment.

The level of category involvement also affects representativeness. Not all products command enough attention to be topic of conversation in social media, unless there is something that surprises or annoys the public (e.g. banks imposing debit card fees).

Then there is the issue of how we separate unpaid opinions from unpaid opinions, which are becoming more common.

APPLICATIONS

For the moment, until better text analytic tools allowing more accurate data cleaning and other tools that would help identify who is behind the opinions, are available, I think social media content should be considered as a tool in the qualitative research arsenal to explore and sometimes dig deeper in certain issues. Among its applications, social media content can be used to explore:

• Likes and dislikes about a product, brand or company
• Language used to talk about needs, expectations, barriers
• Problems with products, customer services that need a quick response
• Topics of interest within a product category
• Perceptions about a brand and its competitors

Social media content can be a source of rich insights, but we should be aware of its limitations and do not equal access to large amounts of text to quantitative research.

A client recently asked for advice about usability testing approaches. Her internal client wanted to do a traditional usability lab test, while she was wondering if a quantitative online usability approach was a better fit.

This question got me thinking about the advantages and disadvantages of each approach. As usual, no research method is perfect and each has his strengths and weaknesses. There are many tools available to do quantitative usability testing online faster and cheaper. Although these can also provide qualitative data, usability testing in the lab environment can give you a deeper layer of insights into user behavior on specific issues.

The table below shows what I have learned about usability testing, after managing two usability labs in my past life as a corporate market researcher.

Each usability testing approach has a purpose and complement each other. If timing and budget permit, you should use both. If you can only afford one or the other, think hard about the objectives and the type of decision you will make based on the results.

For detecting major problems or understanding at a deeper level why people find certain tasks difficult to do or what their expectations are, a qualitative approach is a more appropriate. On the other hand, if you need to make a decision about major changes or a redesign and need to know how big of a problem you have in your hands, or how you compare to your competitors, a quantitative approach would do the trick.

Not long ago I got a call from a potential client asking for research to determine why a recent marketing campaign failed to increase sales, despite a significant increase in awareness.

He had conducted an advertising awareness survey, and the results showed that many in the target audience had noted the advertising and gave it high ratings, but didn’t make a purchase. All possible explanations were merely speculations. He couldn’t pintpoint to any particular cause for this. The main problem was that he wanted to obtain evidence of a cause-effect relationship, but the research design was not appropriate for that.

The main method for causal research is experimentation. In experimental-based research, the causal or independent variables are manipulated in a relatively controlled environment. This means that other variables that may affect the dependent variable (e.g. sales) are controlled and monitored as much as possible.

In this case, the client had conducted the survey and analyzed the data without taking into account the effectiveness of the different marketing collaterals used, its market penetration, competitor activity, and some characteristics of the purchase decision makers. After doing some digging around, we uncovered that in some markets, competitors had launched high frequency advertising campaigns which helped the client indirectly by increasing category awareness, but not his sales. Also, the program targeted customers who were recent buyers and probably didn’t have a need for his products at that particular moment.

Surveys that are not designed as part of an experimental approach may show correlations, but not causality. To really connect the dots between cause and effect, we needed to create an experiment including different renditions of the marketing collaterals, different markets, customers at different stages in the purchase cycle, and different actions taken by competitors.

Experimentation in marketing has traditionally taken the form of standard test markets, in which test market are selected, controlled advertising is put in place, and the product is sold through regular distribution channels. The drawbacks of this type of test are that they can be time consuming, are often expensive and may be difficult to administer.

A more palatable solution is what is called simulated test markets in which individual are selected, exposed to the product or concept (e.g. via actual marketing collaterals), given the opportunity to buy the product in real life and if they buy it, they are asked to evaluate the product and state their repeat purchase intent. The trial and repeat estimates are combined with data about promotions, distribution levels, competitor activity and other relevant pieces of information.

Another possibility can be found in the popular freemium model, adopted by many businesses both in B2B and B2C, which mimics this process to some extent. This model can be used for experimentation and as a rich mine for insights at a low cost. The basic principle is to let people try it and observe what decision they make, after which research can follow to understand what drove their decision, controlling for other variables that may be affecting the outcome.

In short, if you want to understand cause and effect, you need to conduct experiments, which may include surveys as data collection method, but surveys in themselves can’t provide the answer. It is the  experimental design what will lead you to it.