5 Market Research Lessons From Election Polling Miscues

In the NY Times article Pollsters Face Hurdles in Changing Landscape, Ryan Knutson and Aaron Zitner discuss a number of reasons for recent high-profile polling failures, the Brexit vote and the U.S. presidential election.

Why should customer experience (CX) professionals care? Here’s what they say in the article:

The outcome also raises questions about the research businesses rely on to test new products and measure customer behaviors, since many of the same survey methods are used for market research.

The article brings up some good reasons for the poor predictions:

• People are less likely to answer surveys, so it’s harder to get representative samples.
• It’s more difficult and expensive to reach people via cell phones than it was by landline.
• Decision factors are changing. For instance, education level was a more important decision driver in this election than it was in 2012.
• The people who choose to respond to polls don’t fully represent the population.

My take: I’ve been talking about the need to shake-up market research for many years. As a matter of fact, my 2011 post Market Research Needs An Overhaul remains relevant today. All of the issues with recent polling projections are similar to what many companies face when trying to understand their customers. Here are five thoughts on how to prepare your market research efforts for the new realities:

1. Embrace outliers. The traditional approach for dealing with data points that don’t fit a model is to ignore them or discount them as being “outliers.” But these counter-trend pieces of data can be much more than that. They may be a window into an emerging trend or a small signal about a set of customers that your current research is missing. When you see an outlying datapoint, don’t ignore it anymore. Think about what it might be telling you, and what insights you may missing.
2. Always ask “who are we missing?” All research processes, including surveys, are biased in many different ways (see my Latest 9 Recommendations for NPS). You can minimize and address some of the biases, but there’s always the risk that you just don’t see some of them. One of the things you can do is to proactively look for the biases. Always seek to define the populations of people that you are missing or under-representing in your research, whether it’s caused by a demographic or attitudinal blind spot. If you can’t find them, then you haven’t looked hard enough.
3. Listen, don’t just calculate. A lot of my insights about the election came from listening to what people were saying, not from crunching datasets. As the environment around your company changes, you need to spend a lot more time with qualitative, unstructured content. Why? Because structured data collection reflects historical assumptions, and may very well be missing the key variables required to fully understand changing customer attitudes and behaviors.
4. Over-emphasize recency. If you’re building a predictive model, make sure that it is very sensitive to recent data. If you’re mapping out a long-term trend or trying to fit the data to a historical model, it may take a while for you to identify a substantive change in the environment. Even if you don’t change your core model, look at what it says if you significantly over-weight recent data points.
5. Modernize your leadership. The way that organizations can and should use data is one of the shifts that is making traditional management techniques obsolete. That’s why you should adopt what I call Modernize Leadership: Shifting 8 Outdated Management Practices. This requires making a shift to Engage & Empower, Learn & Adjust, Detect & Disseminate, Observe & Improve, Purpose & Values, Strengths & Appreciation, Culture & Behaviors, and Experience & Emotions.

The bottom line: It’s hard to project from the past when the future is changing.