How To Minimize Memory Errors in Surveys

I work hard to minimize memory errors in surveys to gather quality data. Surveys often ask questions about past behaviors, but it is a challenge to do it right.

We assume that respondents can give an accurate account of what they did, how often they did it, or how much time, energy, and money they spent doing it.

Unfortunately, our memory fails more than we want to admit. Respondents, often and unintentionally, make memory errors in surveys which lead to over-or understatements of their actions. 

Factors Affecting Memory in Surveys

Some of the factors that affect our memory in the context of a survey include:

• The type of event we ask about
• Feelings about the event
• Event frequency
• How long ago the event occurred
• How we formulate the questions

All these factors impact our ability to recall particular events. Consequently, we tend to misplace events in time, also known as the telescoping effect. We see this when recent events are perceived as being remote and remote events as being recent.

Type of Memories At Work in Surveys

Research has found a partial explanation for this phenomenon in the way our brain organizes our memories. It stores some as episodic memories. These are specific personal events determined by a particular time and place.

Others become part of our semantic memory. This type of memory includes our general knowledge about the world, not necessarily tied to specific events.

It turns out that the time frame and frequency of events influence the use of one type of memory or the other.

For instance, studies have indicated that while estimating frequency in the occurrence of an event, we tend to use our episodic memory for events that are rare (e.g., car accidents) or are recent (e.g., grocery shopping).

In contrast, we often use our semantic memory when the reference period is long (e.g., past 12 months vs. past 3 months), as we have a harder time keeping the events separate as unique memories over time.

Depending on the type of event and time frame, the telescoping effect may be stronger going forward (distant events perceived as recent) or backward (recent events perceived as remote in time).

Memory Errors Implications For Data Quality

For obvious reasons, this has significant implications for data quality. It is especially important when we use past behaviors within a time frame for sample selection to understand a particular event or behavior.

Survey respondents may give answers about their average behavior over a vaguely defined period when we ask about the last time they purchased something in the previous 12 months. For this type of question, they tend to rely more on their semantic memory than on the episodic memory of the last occasion if this is a frequent behavior.

The opposite may also happen when we ask about an “average” or “typical” behavior.  People are more likely to base their answers on the most recent event which is fresh in their memory. In other words, they end up generalizing one experience to all previous experiences, which or may not be similar.

How To Minimize Memory-Based Errors in Surveys

What can we do to minimize memory-based errors in questions about past behaviors?

• Keep the reference period for the event of interest closer to the time in which the data is collected. We are likely to give more accurate responses to events that happened in the last 7 days than in the past 12 months.
• Connect to meaningful events to anchor the time frame. It can be dates and specific events related to a particular date (e.g., before or after 9/11).
• Provide clues that can be associated with the behavior of interest (e.g., How many times do you drink beer in a week? Think of occasions in which you may drink beer: party, after work, meals, watching TV, with friends, etc.)
• Use “warming-up” questions to trigger memories related to the event of interest (e.g. Do you like beer? What type? How many?)
• Refer to previous answers (e.g., You indicated you have been to the Dr. this month. How many times have you visited your primary Doctor?)
• Use prior records if available (e.g., Our records indicate you called our customer service on mm/dd. How long did you have to wait on the phone?)

Finally, some of these measures are easier to implement than others. Survey length, sample incidence rate, survey tool limitations, lack of information, and other research priorities can get in the way of the appropriate survey design.

When you face these obstacles, do not forget the implications for data quality. Otherwise, you may end up wasting time and money in gathering misleading data.