There’s No Method to Opinion Poll Madness

When I got the e-mail about the new Excite Daily Poll, I immediately climbed on my high horse and ranted in a return message about how “polls� on the Internet were methodologically unsound, dangerously misleading and generally an affront to truth-seekers everywhere.

So, OK, I felt a little sheepish when I saw the poll’s issue of the day on the Excite Web site: Bill Gates’ favorite housewarming gift: a) million-dollar gift certificate to Maids-R-Us; b) super-size thrifty-pack of surge protectors; c) “This man’s home really is a castle!� refrigerator magnet.

Not very funny, but not exactly a threat to civilization.

Still, I’m not ready to dismount just yet. Excite may just be trying to have a little fun, but in doing so it is highlighting a dangerous dimension of the Information Age, as well as one of the most obtuse but widely held assumptions about the Internet.

On the first count, the crux of the problem is the implication, inherent in the word “poll,� that votes or opinions are being registered in a way that means something.

It’s hard to turn around these days without encountering something that’s called a poll. TV stations beckon with “instant polls,� inviting you to call a 900 number and register your opinion. Some newspapers do the same thing. On the Web, hundreds of sites invite you to vote on everything from the merits of Libertarian presidential candidates to whether you favored union or management in the UPS strike.

Yet polls like these don’t mean a thing, for a simple reason: The sample is “self-selected.� In other words, anyone who participates has actively decided to participate; most people, reasonably enough, won’t be bothered. And the people who choose to participate do so, by and large, because they have some kind of particular interest in the issue at hand and thus are by definition not representative of anything other than themselves.

Serious-minded pollsters, including those at major media organizations such as this one, have been battling the growth of “pseudo-polls� for years. The American Assn. for Public Opinion Research puts out a set of guidelines on proper polling practices, and they are elaborate: Producing meaningful results requires careful attention to sample selection, wording of questions, interviewing techniques and statistical analysis methods, among other things. Surveys relying on self-selection are considered worthless by definition.

While pseudo-polls often don’t promise to be something they’re not, it’s asking a lot for people to think through the methodology, or lack thereof.

“If it’s called a poll, folks are not going to look at it very carefully� to determine if it’s real, says Evans Witt, executive director of the Voter News Service, an opinion research consortium. “That confusion ultimately degrades real survey research.�

Often, such confusion is deliberate. Companies use poll results to bolster marketing campaigns and sell products--�four out of five dentists surveyed . . .�--yet there is no reason to believe that such data necessarily mean anything. Sure, the companies usually claim the research is carried out by an independent group, but even in the cases in which the methodology is honest, one hears of the results only if they happen to support the interests of the sponsor.

Just as I was writing the previous paragraph, for example, I got a call from a public relations agency touting a new study of Internet advertising from a consortium of Web sites called the Internet Advertising Bureau. It was a big, elaborate study, and the methodology seems solid. But somehow I’m not surprised that the conclusion is that Internet advertising works even better than previously thought.

Personally, I find even the ever-expanding market research machinery of modern business to be out of control. We’re all being questioned and probed and analyzed constantly to determine our preferences about the most trivial matters, yet the premise behind a lot of this is dubious.

IBM, remember, commissioned a study of the potential market for photocopiers in the 1950s and concluded that only a few thousand could be sold worldwide. Companies, like politicians, spend so much energy determining which way the wind is blowing that they sometimes don’t have anything left for real creativity and innovation.

The Internet, of course, is a great medium for soliciting people’s opinions about any old thing. But Excite, and a lot of Web sites, make the mistake of thinking that just because you can solicit people’s opinions it will be interesting or amusing to do so. It’s interactivity for its own sake.

The desire of Web sites--or TV stations and newspapers, for that matter--to look for new ways to connect with the public is understandable. (Excite invites people not only to vote but to create their own poll questions.) But exploiting the power of the Internet is about creating a meaningful context for interactivity; to solicit the views of anyone who wanders along and then tally up the results is a poor substitute for the admittedly difficult task of building and engaging an audience.

If the opinions in question relate to a serious political, social or business issue, then it’s downright irresponsible to collect them in this fashion. Some have viewed the Internet as a potential medium for “direct democracy,� in which people might electronically decide real political issues--remember Ross Perot?--but self-selected Web polls show just how scary that could be.

So the next time you read or hear that 56% of Americans think Bill Gates is a hero or that 78% plan to eat more chicken next year or that 36% of computer owners hate their PCs, consider the source. In this case, I just made it up.

*

Jonathan Weber ([email protected]) is editor of The Cutting Edge.