Second Edition

Almost every day I find out about a new survey tool. As we all know, the advent of online survey tools has lowered the cost of data collection and made it easier for many internal market research departments on the client side to deploy their own online surveys.

Reduced or non-existent market research budgets continue to be the norm, and since the first publication of this post (3/30/2010) clients are still looking for free survey online tools or very cheap solutions to get the job done. When I ask what tools they use, the most common answer is still SurveyMonkey or Zoomerang.

I used SurveyMonkey a while back when I was on the client side with no budget, but it had so many limitations, when I wanted to do more advanced surveys, that I had to search for another tool. I tried Zoomerang, but I found it to be expensive given the limited features it offered. Besides, I found the interface wasn’t very intuitive. At another company, where I led the research team, the budget allowed me to use VantagePoint, which I then abandoned for Confirmit. The latter is a very advanced tool, but also quite expensive. So my quest continued and then a couple of years ago I stumbled upon SurveyGizmo, which really surprised me with a long list of features I wanted in an online survey tool at a very low cost.

Lately, I have had the chance to use Qualtrics, on behalf of several clients. As I have said below in response to some comments, Qualtrics is a terrific tool. It has tons of features, but it can only be used for free for one project. After that, you need to buy the license, which starts at a higher price point than SurveyGizmo’s equivalent option, the Enterprise edition.

There are many online survey tools that offer a free limited version so people can test the tool and have a feel for it. However, not all free versions are equal. Some give you more than others and you get the chance to really experience how useful the tool can be. This is the case of SurveyGizmo vs. SurveyMonkey. Even at the lowest subscription plan ($19/mo), SurveyGizmo offers many more features than SurveyMonkey.

Since people still ask me what online survey tools they can use for free, I decided to share the free online tool review I made of the three online survey tools that seem to be most popular, on a set of features that would allow me to create from simple to slightly more advanced surveys. By popular demand, I have reviewed the first comparison I did, published on 3/30/2010, and found a few changes and improvements, but nothing substantial.

General Overview
Number of surveys	Unlimited	Unlimited	Unlimited
Number of questions	Unlimited	10	12
Number of responses	250	100	100
Multi-page surveys
Account users	1	1	1
Question types	23	15	15
Graphic survey templates	37	15	21
Real-time reporting
Data exporting
Never delete data
Supports all languages
Price	Free	Free	Free
Question Types
Textbox – Short text
Textbox – Numeric Format Validation
Textbox – Email Format Validation
Essay – Long text
Multiple text fields
Table of text fields
Checkbox
Table of checkboxes
Radio button
Table of radio buttons
Drop-down menu
Tables of dropdown menus
Ranking – Table format
Ranking – Drag & Drop
Ranking – Top X# of items
Ranking – Stars
Constant sum
Continuous sum
Instructions/Text
Image
Video
Contact information form
Date format
Survey Formatting
Customizable font family
Customizable font size
Customizable font color
Customizable background colors
Add/Edit Options in Bulk
Add an open textbox so users can add to their own choice
Change orientation of labels and options
Change position of labels
Editable Next, Submit, and Close buttons text
Optional question numbering
Fully Customizable XHTML/CSS templates
Auto-format as currency, percentage, or a whole number
Editable general error message
Change length, height, and width of textbox or table
Editable survey width
Editable survey after deployment
Report Creation Options
Real-time online report
Graphing – Use a bar graph, pie chart or spreadsheet to view answers
Export Data to Excel, CSV
Browse All Responses Individually
Can change titles and internal report values
Clear all responses
Survey Deployment
Send email using survey tool
Password protection
Set a cutoff date and time
Set maximum response count
Enable IP blocking
Redirect browser
Link to panel providers
Incentive Management tool
Other Survey Options
Make questions required
Randomize the answers to your question
Randomize columns
Reorder Questions
Set how many checkboxes someone can check
Automatically put in a value already collected into the answer field
Upload images
Embed external images and media
Min and Max options for numerical answers
Prevent search engines from indexing survey
Thank You page
Add a “Close” button to Thank You page
Back Button Optional
Progress Bar Optional
Add a unique survey identifier to each link
JavaScript option embed surveys using JavaScript
Embed surveys using an iFrame
Embed surveys into a Webpage
Create hidden variables
Compliance with Section 508 guidelines of the Rehabilitation Act
Export survey to PDF
Export survey to Word
Manual data entry of additional records
Polls
Poll templates	37 + Custom		8 + Custom
Poll Types	Single choice, Image Choice, Ranking		Single choice, Multiple Choice
Poll result sharing
Vote protection	By IP address, by cookies, or none		On/Off
Poll Widget sizing
3rd Party Integration
Open API access
Facebook integration
Twitter integration
MailChimp integration
Affiliate program
Support
Support via email
Support – FAQ, Tutorials

Last update 6/28/2011

There are many other tools that offer a free version, but I found them ever more limited that those shown here. As you can see SurveyGizmo has many more useful features in its free version, although a few are missing. Survey deployment capabilities continue to be a weak area for Survey Gizmo, where SurveyMonkey does better. For example, in SurveyGizmo’s free version I miss the ability to send survey invitations within the tool (you can do it with a paid subscription), but that can be fixed using MailChimp or any other email distribution system. If you use an external system, be aware that in order to send reminders, you have to download the data to check who answered the survey and then extract those from the reminder list. It is a bit of work, but doable. Neither SurveyMonkey or Zoomerang let you download the data without upgrading to a paid subscription.

In this updated comparison, I added Polls, which both SurveyGizmo and Zoomerang offer as a separate survey type, given some specific features they have (voting, sharing, one-question). I find both pretty comparable.

As a professional market researcher I usually need very robust tools, so free online survey tools usually fall short on providing features needed to develop complex surveys. Notice that none of these 3 free versions offer skip logic, but if you only need to deploy linear surveys without having to skip questions or reroute respondents, and don’t want or can’t spent a dime, I recommend you give SurveyGizmo a try. If you have some change to spare, I recommend you use one of SurveyGizmo paid subscription plans, which don’t require a long term contract obligation and have many more features at a very low price.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.

If you go by the media buzz, it seem that everybody has a tablet these days, but  a recent Forrester survey shows that in Q1 2011, about 5% of US online consumers have a tablet, although other surveys cite higher numbers and shipments are expected to be even higher in the near future. What it is interesting is to learn how tablets are being used by consumers. Here are some numbers presented recently at the 2011 Market Research Association Annual Conference in Washington DC by Forrester’s James McQuivery.

One of the presentations I enjoyed the most at the recent 2011 Market Research Annual Conference in Washington DC was the one by Barry Blyn from ESPN. He provided superb examples of how market research should be implemented and add value to an organization.

First, Blyn made an important distinction between Measurement and Insights. Measurement tells you what people did, while Insights tells you why they did it and how to get them to act in the future.  In my view this is where market researchers can add most value.

ESPN’s research efforts are led by these guiding principles:

1. Research projects are about insights working more like weathervanes than speed cops. It is more important to know where the business is going (weathervane) than measuring how fast things went by (speed cop).
2. Research projects are led by neither quantitative nor qualitative work. They work together as peanut butter and jelly. Results from any research method are presented together to answer the business problem.

According to Blyn, ESPN has become fanatical about listening to their audience. In 3 years they have conducted more than 400 in-depth interviews all over the country and 15,000 surveys. In search for insights they have been combining traditional and innovative types of research with consumers trying to get at the heart of the business challenges ESPN faces. Blyn presented two examples of how these guidelines are implemented:

• Brand Health Tracking (Crusade): This is a brand tracking program which is seen as an orchestra where different research methods are thought of as instruments and are combined to reach the ultimate goal: monitor ESPSN’s brand strengths and hunt for weaknesses making sure ESPN is always in touch with fans’ perceptions about the ESPN brand. In this program, ESPN combines qualitative and quantitative methods such as:
  • RDD telephone interviews with 1,500 adults 12+
  • Quarterly online surveys with 1,600
  • Backyard barbecues with fathers and sons (group discussions)
  • Conflict groups with ESPN fans debating ESPN detractors
  • “College campus crawls” (Group discussions with students about sports over beers and wings at a college bar)
  • Word of mouth/social media  monitoring
  • Attending a baseball games with fans
  • Brand Eulogy (Imagine ESPN has passed away and deliver a eulogy)

• Deprivation study: This is by far the most provocative research approach Blyn presented. In this case, they paid $400 to 60 avid sport fans (30 who only used ESPN TV and 30 who use ESPN TV and other platforms to get sport news) for abstaining from watching ESPN during football season. The main objectives were to:
  • Get vivid illustrations of how consumers felt about the brand
  • Identify where fans were going when they couldn’t watch ESPN TV (identify content “blindspots”)
  • Build a five-tool player, meaning being a brand that can do it all for everybody in different platforms

In this study, ESPN allowed participants to provide feedback through different channels: video and audio journals, focus groups (“therapy” sessions), before and after surveys, and in-depth interviews. From this research it became clear that ESPN needed to align its different brand properties from a fan centric point of view.

In a time when many believe the market research industry have just missed the train, the ESPN case shows that to make research relevant to organizations today, market researchers need to:

1. Become more insight-centric. Bring the insights into the spotlight and leave the research methods used in the background
2. Provide forward-looking insights
3. Use hybrid research approaches and triangulate results

I think we can all do that.

In one of the keynote speaker sessions at the recent 2011 Market Research Association Annual Conference, in Washington DC, a delightfully loud and dynamic banter went between Marshall Toplansky from Core Strategies and Bill Neal from SDR Consulting trying to answer questions such as:

Is marketing research really able to deliver on the critical needs of today’s (and tomorrow’s) enterprise?

Most of the discussion revolved around the pros and cons of using social media in market research. Toplansky  focused on the need for a real-time flow of data, which social media can provide, while Neal called researchers to be the voice of the customer and make sure the data we gather, no matter the channel, represents the target market.

According to Toplansky, the type of information the traditional market research provides is irrelevant to decision makers. Big corporations, which are responsible for 85% of the market research expenditure, are funding real-time, continuous flow of information on which they can make decisions on (e.g. lead generation, sales, promotions, competitors’ impact, etc.), which has been made possible by technology. For Toplansky, it is about harnessing technology and providing daily information at a cost that is equivalent to years of doing tracking research.

Neal, on the other hand, argued that our role as researchers is to find out what is going on in the market place and why. According to Neal, the” why” is not being addressed by the new technology.  He is also an ardent supporter of sample representativeness. No matter how large the amount of information we may be able to collect via social media, we always have to ask if it is representative of our target market, which Neal claims, it is unlikely to be.

Our role as market researchers is, said Neal, to be the voice of the customer, but unfortunately market researchers don’t have a place in the C-suite. Most researchers work for the CMO, but the money is managed by the CFO. This is often reflected in research guided by the “I have to have it now” mentality which often leads company astray. Unfortunately, the people who are users of consumer data are not able to judge its quality, said Neal. Corporate researchers have to act as the guardians of data quality.

For Toplansky, representativeness is more about finding consumers who are engaged with a particular brand or product category than demographic representation of the population. Companies don’t care about the general population, but about the consumers who will buy their products.

Since many companies are run quarter to quarter, argued Toplansky, it is about building the business around empirically read mass observations, and correlating sales numbers, channel-through sales numbers and other relevant metrics. The reason we are not in the C-suite, said Toplansky, is because researchers don’t speak that financial language. We fail to translate consumers’ preferences and understanding into a continuous flow of data needed for decision making.

For Toplansky, traditional market research is too slow and expensive, so finance managers default to the information they have at hand under the mantra that “some data is better than no data,” and “it doesn’t have to be perfect.” This attitude is very detrimental to the perceived value of the market research function.

Although both speakers sounded like coming from opposite points of views, I found that their arguments complemented each other:

• Social media can provide invaluable consumer data, but we need to make sure that what we hear in social media represents our target market, that is the consumers we are after, which may or may not be representative of the general population.

• Social media research can be a source of valuable insights to explain the results from traditional research and even help to design better data collection tools (i.e. surveys, discussion guides, IDIs).

• Social media can help to explain the “why” behind sales and market share data. This requires better text analytics tools that allow us to dig deeper in the content provided by social media.

• Triangulation of different data sources, including social media, is needed to translate consumer research into financial terms and increase market research’s predictive power. There is not a research method that can do this on its own.

• Market researchers need to find ways to provide more timely and relevant information to support decision making.

• Market researchers would benefit from learning more about marketing and the company’s financials to be able to make market research more actionable.

In my view, the market research industry has not missed the train. The train has just arrived and we are all trying to figure out how to hop onto the wagon without leaving valuable knowledge behind at the same time as we inspect how this new train works and where it can take us.

One of the workshop sessions I attended on the first day of the 2011 Marketing Research Association (MRA) Annual Conference, in Washington DC., was about mixed data collection modes. One of the hottest topic in our industry today.

ROUND I

Annie Petit from Conversitions/Research Now showed examples of how online surveys, text messaging and social media research can be used in combination to get the best out of each. I totally agreed with Petit when she said there are not perfect methods. We have to use all methods that are available and get the best out of each.

For example, surveys are great for frequencies, and representative demographics; text messaging is good for on-spot live behavior, to check what people are seeing, eating, drinking right now; and “social mediaresearch is like doing a 10-hour survey,” which can give us tons of data, from thousands and millions of people on any one topic. We can use them all together. Examples:

• Use social media research to identify the best options to put on a grid question (e.g. list of the most popular restaurants)
• Use sharing of media research with respondents as an incentive
• Combine social media and survey with diaries via text messaging (SMS) to understand behavior
• Invite people to a SMS survey from an online survey
• Use social media research to do a deep-dive into the results from an online survey or a SMS survey

ROUND II

In the same spirit, Stephen Murrill, from Meta Research presented  a case study of the mixed mode approach used for the California State Fair, which was interested in understanding:

• Satisfaction drivers
• Barriers to attendance
• How to motivate families to maintain the tradition of coming to the Fair
• What activities may attract teenagers

The Fair usually used exit surveys to measure satisfaction with the event, but now they needed to talk to teenagers and Non-Fair-goers for whom exit surveys were not an option. The solution was to combine data collection on-site using iPads to get feedback from Fair-goers, SMS surveys to reach teenagers that came to the Fair, and phone and online surveys to reach Non-Fair-goers.

ROUND III

Sima Vasa from Paradigm Sample and Leslie Townsend, from Kinesis Survey Technologies, joined forces to present a case study where the primary data collection mode was mobile surveys. The problem was presented by a CPG client wanting to gain insights about consumers of convenience stores (C-stores), particularly Millennials (18 – 34). The goal was to create a consumer panel of C-store shoppers and keep them engage to learn more about these consumers over time.

Since this age group often doesn’t participate in online surveys, the solution was to develop a customized mobile panel application, using the Kinesis platform, that users would download to their cell phones. This application allowed to send surveys and capture the C-store experience of this age group. The application was optimized for many different devices and detected which device were used, which allowed to target the surveys appropriately, so if a longer survey was sent, they could advise users to take the survey on a PC instead of on their cell phones.

ROUND IV

Finally, Rick Kelly from Opinionology/Survey Sampling, talked about a study done for the Tour of Utah cycling race to understand the impact of advertising during the race on purchase intent. Participants were given the chance to participate in the study  using one of three modes: a mobile survey (SMS), an online surveys, or an IVR survey. The mobile survey had 12 questions, while the other two modes had 20 questions. The survey was open during a whole month, achieving 10% response rate.  Among the results presented, we learned that the SMS survey was the most common mode used buy those answering the survey at the event, which were also younger respondents. Most of those who answered the survey online did so within 24 hrs, and participants over 55 were overrepresented among IVR survey respondents. The results also showed an increased in purchase intent over the time the survey was open, for which there was no clear explanation.  Of all the four presentations, this one left me wanting for more in order to understand how the different modes actually compared.

KEY TAKEAWAYS

Although I didn’t learned anything totally new from this workshop, the speakers made a good case in favor of using mixed data collection modes. However, we should first define what we need to know and who we need to reach and then decide which data collection modes are more appropriate for the research objectives.

Petit showed good examples of how we can incorporate social media research to support more traditional research methods. Use it to refine survey questions, dig deeper about survey results, recruit respondents or provide incentives.

Murrill showed a great example of how different modes can be used to reach different target markets, while Vasa and Townsend showed how new nechnologies can help ups gain insights about hard-to-reach market segments.

What I missed was a discussion about the major issue that combining different data collection mode has, namely,  the measurement error introduced by different data collection modes as they have an impact on how people answer questions. Maybe, next year.