After posting the comparison of three popular online survey tools’ free versions, many have requested a similar analysis of paid online tools. It seems there is a new tool coming up every day and comparing them all would be a rather difficult task. I decided to compare three online survey tools I have used most recently for different projects.

I have been using SurveyGizmo for a while now. As I said in my previous post, it is a great value given the number of features it offers at a very affordable price. The tool has been improving with time so it was a natural choice for comparison purposes.

Not long ago, I saw a demo of Beacon by Decipher and was pleasantly surprised by the advanced features if offered and the business model that supports it. Most online survey tools require either a monthly or annual subscription, but Beacon does not. You only need to pay for completed surveys. I decided to give it a shot and used it to program a pretty complex survey. I had the chance to become very familiar with the tool, and thought it was worth including it in this comparison.

Finally, since my above mentioned post about free online survey tools, I have had many requests asking me to review Qualtrics, which is also becoming quite popular. I happened to use it in two recent projects with very different requirements, which gave me the opportunity to test it in depth.

The list below includes some of the most often-used features in moderate to advanced survey programming and, as you can see below, all three online survey tools are pretty powerful and comparable in many aspects. That said, none of these online survey tool is perfect, and they all would benefit from continuous improvement in capabilities and usability. Most of the time they do the same thing in different ways that have an impact on  users’ learning curve.

General Overview
Number of surveys	Unlimited	Unlimited	Depends on license
Number of questions	Unlimited	Unlimited	Unlimited
Number of responses	Unlimited	Unlimited	Depends on license
Number of email invitations	Up to 500k	Unlimited	Depends on license
Multi-page surveys
Account users	Up to 20 users	Unlimited	Depends on license
Question types	32	22	38
Graphic survey templates	37	9	55
Real-time reporting
Data exporting
Data deletion	Never	Never	If account is closed
Supports all languages
Price	$159/mo. 10% Off with annual subscription	No subscription needed. Pay $0.50/complete	From Free trial. $5,000/year
Question Types
Textbox – Short text
Textbox – Numeric Format Validation
Textbox – Email Format Validation
Essay – Long text
Multiple text fields
Table of text fields
Checkbox
Table of checkboxes
Radio button
Table of radio buttons
Bipolar, semantic ale table
Drop-down menu
Tables of dropdown menus
Ranking – Table format
Ranking – Drag & Drop
Ranking – Top X# of items
Ranking – Stars
Constant sum
Continuous sum
Constant sum table
Descriptive Text/Image
Image choice (radio button/checkbox)
Video		Hosted for a fee
Contact information form
Date format
Heatmap
Image/Text Highlighting
Slider
Card Sorting
Stack Sorting
Question/Page timer
File upload
Captcha
Drill down
Combination of different question types in one table
Survey Formatting
Customizable font family
Customizable font size
Customizable font color
Customizable background colors
Add/Edit Options in Bulk
Change orientation of labels and options
Change position of labels
Editable Next, Submit, and Close buttons text
Editable Next, Submit, and Close buttons graphic elements (color, size, borders)
Optional question numbering
Fully Customizable XHTML/CSS templates
Can create own CSS templates
Auto-format as currency, percentage, or a whole number
Editable error messages
Change length, height, and width of textbox or table
Editable survey width
Editable survey after deployment
Min and Max options for numerical answers
Reorder Questions
Randomize the answers
Randomize columns in a table/matrix question
Randomize questions within a page
Randomize pages
Set how many checkboxes someone can check
Optional survey tool’s logo			Depends on license
Add your own logo /White label			Requires custom development for a fee
Create max word size
Separate answer titles and reporting values
Question grouping in blocks
Logic
Question-based skip logic
Page-based skip logic
Unconditional skip logic
Show and hide questions based on logic
Question trigger on the same page
Pipe answer titles or reporting values from a question into another
Repeat questions based on answers to previous questions
Repeat pages based on answers to previous questions
Add values from a URL variable as question options
Add custom script / Javascript
Quiz score with tally
Quiz score with pass/fail
Make questions required
Make the questions soft-required
Random assignment of respondents to different paths/cells/blocks
Quota-based logic
Loops
Report Creation Options
Real-time online report
Crosstabs	Counts, Percentages	Counts, Percentages, Means, Top 2 Box	Counts, Percentages
Statistical Testing		T-Test, Test of Proportions	Chi-Square
Graphing – Use a bar graph, pie chart or spreadsheet to view answers
Export data to Excel, CSV
Export data to SPSS
Browse all responses individually
Can change titles and internal report values
Clear all responses
Share results with others via a link with or without a password		Only with password by creating user accounts
Share results with all respondents by including a link on the Thank You Page
Add a sharing/bookmarking button
Export report to PDF
Export report to Word
Export report to Excel
Export report to E-mail
Export report to Power Point
Survey Deployment
Send email using survey tool
Set a cutoff date and time
Set maximum response count
Enable IP blocking
Redirect browser
Link to panel providers
Pass data through survey link
Add a unique survey identifier to each link
Embed surveys into a Webpage
Embed surveys using an iFrame
Collect data in offline mode
Collect data in Kiosk mode
Convert link into QR code
JavaScript option embed surveys using JavaScript
Question-based quotas
Close survey based on total quota
Remember respondents’ place in survey if completion interrupted
Other Survey Options
Pipe in values from answered questions into later questions’ answers
Pipe in values from answered questions into later questions’ text
Upload images to internal library
Embed images and media via external link
Prevent search engines from indexing survey
Thank You page
Add a “Close” button to Thank You page
Back Button Optional
Progress Bar Optional
Create hidden variables
Compliance with Section 508 guidelines of the Rehabilitation Act
Export survey to PDF
Export survey to Word
Import SurveyMonkey surveys
Create mobile-compatible surveys
Import data from other data sources
Polls
Poll templates	37 + Custom		Customizable CSS
Poll Types	Single choice, Image Choice, Ranking		Single choice, Multiple Choice
Poll result sharing
Vote protection	By IP address, by cookies, or none		On/Off
Poll Widget sizing
3rd Party Integration
Open API access			Available for certain accounts
Facebook integration
Twitter integration
MailChimp integration
MailChimp integration
iModerate integration
ClickTale integration
ExactTarget Surveys integration
Freshbooks Surveys integration
Jive Surveys integration
Eventbrite integration
GoDaddy integration
Twilio SMS integration
Cint Panel integration
WhatUsersDo Testing integration
Support
Support via email
Support via phone
Support – FAQ, Tutorials

I find SurveyGizmo to be the most user-friendly and you can do pretty advanced programming with it. They have been adding new question types and integration partners. However, the data export to SPSS and result reporting feature need improvement. For some reason multiple choice questions get some random variable names that don’t match the variable name specified in the survey, when they get exported to SPSS. Variable names don’t show up in the online reports either. Also, I wished they included statistical testing in their crosstabs.

Qualtrics is the second most user-friendly survey tool of the three. Some things are not obvious. I like the large number of question types. Many are variations of standard questions with horizontal and vertical layouts, with and without images. Among the question types, you will find Constant sum tables, Heatmaps and Hotspots (The latter is pretty useful in concept testing). They recently added a new reporting tool, which is still in beta. I like that you can add banner points to the crosstabs and drill down through different question layers, unfortunately they only give you counts and the reporting page freezes a lot.

Beacon is a powerful tool, but the learning curve is a bit steep. Thankfully, they have a very responsive technical support, and I was able to get answers to my long list of questions and issues, very quickly. What I liked the most about Beacon is the flexibility it offers to create very complex skip patterns and validations. I also liked its crosstabs. They include counts, percentages, means and Top 2 Box with statistical testing. I was mostly bothered by a somewhat cumbersome process for previewing and testing the survey, which I think is something basic that should be easier to do. The tool seems to have been developed by and for programmers since the environment is not very intuitive for neophytes. You can spend a lot of time trying to figure out how certain basic features work.

Regarding cost, each online survey tool uses a different model. SurveyGizmo’s subscription can be monthly or annual (the latter gives you a 10% discount). Beacon doesn’t require a subscription, but charges $0.50/complete. This may be a good deal compared to SurveyGizmo as long as your project’s sample or the total sample across different projects is 300 or less. If you go over 300 respondents, SurveyGizmo would be a more economical alternative with unlimited completed surveys.

Qualtrics is the most expensive of the three. The number of completed surveys will depend on license cost and you have to commit for a whole year, so consider your research volume before making such commitment. However, if you have need of some of their unique question types (e.g. Hotspot), it may be worth it.

As I said before, no online survey tool is perfect. All have quirks when you get into the details. The decision to use one or the other should be based on how well they meet the needs of the project you are working on and what the budget is.

The recent Big (D)esign conference in Dallas, which gathered many in the graphic design industry, website developers, and gamers, dedicated a track to usability. This track included a presentation by Ryan Smith from Qualtrics. Smith discussed how survey tools are evolving to provide a better user experience and match what’s happening in other areas where technology is setting the pace. Using examples from Qualtrics, he demonstrated how survey tools are allowing us to be:

• More interactive: Different questions types have emerged in an attempt provide a more interactive experience to respondents, engage them, increase response rate and response accuracy. Examples include:
  • Rating questions using sliders, images instead of numbers of text and gauges, among others

  

  • Constant sum questions using draggable bars.

  

  • Text Highlight and Heat Maps.

  

• More intelligent: Technology allows us now to do on-the-fly analysis to personalize the experience and integrate survey results with other data sources on a timely manner. We can use advanced skip logic to present relevant questions and capture session information to improve data quality (e.g. removing speeders).

• Better subject selection: Online surveys can be distributed in different ways offer better access to the target audience. Surveys can be deployed using pop-up, pop-under, pop-over, feedback links, embedded links, and link redirects. This allow us to go where the respondents are.

• More immediate: Organizations can quickly conduct surveys with internal resources (DIY research). Survey results can be monitored since the moment the survey is launched. Online reporting capabilities allow us to get updated results continuously. The extent to which we limit or extend result sharing is under our control.

There is no doubt that technology has made it possible to create more engaging surveys, clean the data on-the-fly, facilitate access to surveys, and provide quick results. However, before you jump on the wagon of the cool question types, consider these issues:

• Need: Determine which interactive features are necessary, which ones are more of a novelty, which ones do not help respondents directly to complete the survey, and which ones increase respondents’ burden.

• Culture: When introducing graphic elements like images (e.g. smilies) or graphic representations of objects (thermometers, gauges, etc.), consider who your target audience is and what cultural barriers may influence reactions to these questions. This is especially true in international studies.

• Expectations: People are used to traditional question types, so it may be unclear what it is expected from them. Clear instructions about the scale meaning and actions the respondent need to take are recommended. This may not be a problem, once these question formats become the norm, but for now, we need to ensure respondents know how to interact with the questions.

• Impact on completion rates and data quality. The time needed to get familiar with the new question format, process the instructions (even if it takes a few seconds) and question complexity may affect completion rates and data quality. Research on this is a mixed bag, so to be on the safe side, test the questions with a few members of the target audience before a full launch.

• Reaching people with disabilities: The new question types are often not 508 compliant, making it difficult for people with disabilities to participate in surveys. If you need to reach respondents that may have disabilities, stick to the traditional formats.

Second Edition

Almost every day I find out about a new survey tool. As we all know, the advent of online survey tools has lowered the cost of data collection and made it easier for many internal market research departments on the client side to deploy their own online surveys.

Reduced or non-existent market research budgets continue to be the norm, and since the first publication of this post (3/30/2010) clients are still looking for free survey online tools or very cheap solutions to get the job done. When I ask what tools they use, the most common answer is still SurveyMonkey or Zoomerang.

I used SurveyMonkey a while back when I was on the client side with no budget, but it had so many limitations, when I wanted to do more advanced surveys, that I had to search for another tool. I tried Zoomerang, but I found it to be expensive given the limited features it offered. Besides, I found the interface wasn’t very intuitive. At another company, where I led the research team, the budget allowed me to use VantagePoint, which I then abandoned for Confirmit. The latter is a very advanced tool, but also quite expensive. So my quest continued and then a couple of years ago I stumbled upon SurveyGizmo, which really surprised me with a long list of features I wanted in an online survey tool at a very low cost.

Lately, I have had the chance to use Qualtrics, on behalf of several clients. As I have said below in response to some comments, Qualtrics is a terrific tool. It has tons of features, but it can only be used for free for one project. After that, you need to buy the license, which starts at a higher price point than SurveyGizmo’s equivalent option, the Enterprise edition.

There are many online survey tools that offer a free limited version so people can test the tool and have a feel for it. However, not all free versions are equal. Some give you more than others and you get the chance to really experience how useful the tool can be. This is the case of SurveyGizmo vs. SurveyMonkey. Even at the lowest subscription plan ($19/mo), SurveyGizmo offers many more features than SurveyMonkey.

Since people still ask me what online survey tools they can use for free, I decided to share the free online tool review I made of the three online survey tools that seem to be most popular, on a set of features that would allow me to create from simple to slightly more advanced surveys. By popular demand, I have reviewed the first comparison I did, published on 3/30/2010, and found a few changes and improvements, but nothing substantial.

General Overview
Number of surveys	Unlimited	Unlimited	Unlimited
Number of questions	Unlimited	10	12
Number of responses	250	100	100
Multi-page surveys
Account users	1	1	1
Question types	23	15	15
Graphic survey templates	37	15	21
Real-time reporting
Data exporting
Never delete data
Supports all languages
Price	Free	Free	Free
Question Types
Textbox – Short text
Textbox – Numeric Format Validation
Textbox – Email Format Validation
Essay – Long text
Multiple text fields
Table of text fields
Checkbox
Table of checkboxes
Radio button
Table of radio buttons
Drop-down menu
Tables of dropdown menus
Ranking – Table format
Ranking – Drag & Drop
Ranking – Top X# of items
Ranking – Stars
Constant sum
Continuous sum
Instructions/Text
Image
Video
Contact information form
Date format
Survey Formatting
Customizable font family
Customizable font size
Customizable font color
Customizable background colors
Add/Edit Options in Bulk
Add an open textbox so users can add to their own choice
Change orientation of labels and options
Change position of labels
Editable Next, Submit, and Close buttons text
Optional question numbering
Fully Customizable XHTML/CSS templates
Auto-format as currency, percentage, or a whole number
Editable general error message
Change length, height, and width of textbox or table
Editable survey width
Editable survey after deployment
Report Creation Options
Real-time online report
Graphing – Use a bar graph, pie chart or spreadsheet to view answers
Export Data to Excel, CSV
Browse All Responses Individually
Can change titles and internal report values
Clear all responses
Survey Deployment
Send email using survey tool
Password protection
Set a cutoff date and time
Set maximum response count
Enable IP blocking
Redirect browser
Link to panel providers
Incentive Management tool
Other Survey Options
Make questions required
Randomize the answers to your question
Randomize columns
Reorder Questions
Set how many checkboxes someone can check
Automatically put in a value already collected into the answer field
Upload images
Embed external images and media
Min and Max options for numerical answers
Prevent search engines from indexing survey
Thank You page
Add a “Close” button to Thank You page
Back Button Optional
Progress Bar Optional
Add a unique survey identifier to each link
JavaScript option embed surveys using JavaScript
Embed surveys using an iFrame
Embed surveys into a Webpage
Create hidden variables
Compliance with Section 508 guidelines of the Rehabilitation Act
Export survey to PDF
Export survey to Word
Manual data entry of additional records
Polls
Poll templates	37 + Custom		8 + Custom
Poll Types	Single choice, Image Choice, Ranking		Single choice, Multiple Choice
Poll result sharing
Vote protection	By IP address, by cookies, or none		On/Off
Poll Widget sizing
3rd Party Integration
Open API access
Facebook integration
Twitter integration
MailChimp integration
Affiliate program
Support
Support via email
Support – FAQ, Tutorials

Last update 6/28/2011

There are many other tools that offer a free version, but I found them ever more limited that those shown here. As you can see SurveyGizmo has many more useful features in its free version, although a few are missing. Survey deployment capabilities continue to be a weak area for Survey Gizmo, where SurveyMonkey does better. For example, in SurveyGizmo’s free version I miss the ability to send survey invitations within the tool (you can do it with a paid subscription), but that can be fixed using MailChimp or any other email distribution system. If you use an external system, be aware that in order to send reminders, you have to download the data to check who answered the survey and then extract those from the reminder list. It is a bit of work, but doable. Neither SurveyMonkey or Zoomerang let you download the data without upgrading to a paid subscription.

In this updated comparison, I added Polls, which both SurveyGizmo and Zoomerang offer as a separate survey type, given some specific features they have (voting, sharing, one-question). I find both pretty comparable.

As a professional market researcher I usually need very robust tools, so free online survey tools usually fall short on providing features needed to develop complex surveys. Notice that none of these 3 free versions offer skip logic, but if you only need to deploy linear surveys without having to skip questions or reroute respondents, and don’t want or can’t spent a dime, I recommend you give SurveyGizmo a try. If you have some change to spare, I recommend you use one of SurveyGizmo paid subscription plans, which don’t require a long term contract obligation and have many more features at a very low price.

To learn more about our consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact us.