We depend on survey research for the information that makes economy tick and supports democratic processes, but response rates are declining. What can we do about it?
Business and government and depend on government surveys to for reliable economic statistics on unemployment, health insurance, inflation, and poverty. Businesses also depend on survey research to help them discover new opportunities, understand consumer motivations, configure and price new products and services, and design and target messages to best prospects. Survey research is also an essential part of our democratic process: Social science tells us that people are deeply affected by other people’s opinion, or their perception of them. Elizabeth Noelle-Neumann’s Spiral of Silence explains how the Nazis used public ignorance on public opinion to stifle dissent. It is no accident that dictatorships do not allow public polling.
If asked why survey response rates are declining, most researchers will cite the large and growing number of mobile-phone-only households, the availability of voice mail to screen calls, the clutter that clogs so many online in-boxes, and public concerns about privacy. All of these are serious problems that make it more difficult and expensive to conduct quality survey research.
But all of these problems can be at least partially mitigated by four factors: time, money, and the combination of expertise and empathy.
Time. Today, most researchers, particularly market researchers, are under tremendous pressure to conduct studies as quickly as possible and, often, to meet unrealistic deadlines. A few years ago, a client representing one of the largest corporations in the world told CASRO members that he expected online survey turnaround to take hours not days. Some of us protested, saying that pulling a survey out of the field after just a few hours would not only assure a low response but would also mean that the pool of respondents would most certainly be non-representative. He was not moved by our arguments. By contrast, clients who allow a reasonable amount of time for data collection are rewarded by higher response rates and more valid results. If you doubt this, read Public Opinion Quarterly or attend the annual conference of the American Association for Public Opinion Research. Many of the papers presented prove the point that it is still possible to get respectable response rates if you make the effort and allow enough time.
Money. Most research budgets are not sufficient. Almost everyone working in the field of survey research today has had to make the choice of cutting research quality or blowing the budget. Since blowing the budget is not good for one’s career or financial health, it is usually necessary to cut quality.
It is no coincidence that well-funded studies get above-average response rates since having more money to spend allows one to make a greater effort to reach respondents and convince them to participate. It also allows one to utilize the services of higher-level people to design the survey and manage quality control, which brings us to the final point, a combination of expertise and empathy.
Expertise/Empathy. Respondents often complain about long, boring, and repetitive surveys and questions that are difficult to answer or are too personal. This makes me believe that many don’t participate because they have been subjected to too many inept and/or abusive surveys prepared by people with no background in the art and science of survey research and with little appreciation for the fact that respondents are volunteers.
Today, anyone can conduct an online survey at minimal cost for data collection. The results may be worthless or, worse, misleading, but it can be done. As a result, most people are subjected to a continual stream of email invitations. I decided to click on all the invitations I got for a few weeks and was dismayed by the results. A few examples:
– A customer satisfaction survey from a major financial services company that went into exquisite detail about every experience with their services I had ever had and asked me to evaluate each experience on the basis of attributes I had never thought about. I finally succumbed to boredom and fatigue and broke off.
– A spate of invitations asking me to evaluate online purchases I had made from a department store. Several invitations arrived before the packages were delivered and way before I could have had time to use the product.
– An invitation for a branding study that asked a long list of attributes for every competitor I said I was aware of. It was very difficult to provide responses because many of the attributes did not seem to apply to the product class and there was no “no opinion” or “not sure” option. Since I had to provide an answer to proceed, I just guessed. Garbage in, garbage out!
In short, were I not a researcher, I surely would have been dissuaded from taking surveys in the future. And I doubt that the survey sponsors got valuable information from their misguided efforts.
Back in the heyday of survey research, when response rates hovered around 80 percent, being asked to participate in a survey was a rare event so most people were eager to do so. Surveys, usually conducted by telephone – or occasionally in-person – were expensive to mount so those funding the research took care to hire the most knowledgeable professionals to develop questionnaires. Then, senior researchers monitored the initial interviews to make sure the questions were understood by respondents and working as intended. In other words, they knew their business (expertise) and worked hard to understand what made their respondents tick (empathy).
Six Things You Can Do To Improve Survey Research Quality
1. Become an advocate for respecting and trying to understand respondents. If the survey doesn’t work for respondents, it doesn’t work!
2. Understand that designing and conducting reliable and actionable research is not easy. It can’t be assigned to the most inexperienced person in the office. Nor can a template for your branding or customer satisfaction survey be downloaded from your software vendor’s website.
3. Get out of your organizational mindset and focus on the respondent’s frame of reference when you write survey questions. Conduct cognitive interviews and/or qualitative research to help you better understand they way they think and talk.
4. Don’t be afraid of change. Understand that habit is not a good reason to continue research that is no longer working, if it ever did.
5. Fight for budgets that are adequate to support quality survey research. That includes writing to your Member of Congress asking him or her to vote for more funding for the census and other government surveys.)
6. Hire expert consultants if you don’t have the relevant expertise in your own organization.
7. Keep the survey length to 15 minutes or less.
8. Watch for my next post: Getting More Out of Your Branding Surveys Without Antagonizing Respondents.