YouGov're Having A Laugh

A few weeks back I wrote about the surveys-of-2,000-people which form a growing proportion of British news stories. Suppose you're a company or activist group. You commission a survey of 2,000 people, and ask them some questions vaguely relating to your product or cause. You pick the most interesting results, write them up into a publication-ready press release, and send it to journalists. There's a good chance that your press release will appear, with minor alterations, as a news story in the British media. Like these articles (BBC, Daily Telegraph), which bear a striking similarity to this press release.

Which is good news for you. You get your name in the papers, and it doesn't even look like advertising. Journalists get column inches for very little work, and the pollsters who conducted the survey get publicity too (and your money). Everyone wins, except the public, who end up bombarded with usually meaningless statistics in the guise of "research".

The survey doesn't have to be of 2,000 people, but that's the norm. This is because this is the size of the samples used by YouGov PLC, who are responsible for (at least it seems to me) the majority of these things.

YouGov polls are everywhere. I'd always assumed that they were telephone surveys of a random sample of the British population. I assumed that because I thought that they were meant to be representative. How naïve.

In fact YouGov polls work like this: you sign up as a panelist, online, which takes two minutes. You then occasionally get e-mails inviting you to do surveys. If you do one, you get 50p credit. When you've got £50 they send you a cheque. It's a great way of making cash online, according to websites about making cash online. Sign up here to get in on the action.

So, the participants in all YouGov polls are not random people but are both self-selected and financially motivated. Many of them will be just doing it for the cash, in which case they will be trying to answer the survey as quickly as possible.

Worst, the panelists are not representative of the British population - they consist of people who use the internet, have heard about YouGov, and chose to participate. YouGov say they have 200,000 users, out of 60 million British people.

Every day, YouGov sends a survey to a certain sample of their users which collects 2,000 responses. They call this the "Omnibus" poll. It costs £500 to commission one, according to the leaflet. That's chump change in advertising terms; I don't know how much it would cost to run an ad in one or more newspapers, but it would be much more. With a YouGov poll you might even get onto bbc.co.uk, which doesn't do paid advertising at all. You can see why it's so popular.

YouGov defend their methodology against criticisms. Their main argument is that their approach has a track record of being accurate in predicting the outcome of British elections. But political polling is unique. Politics is one of the few things that most people have strong opinions about. And elections are just big polls, after all. Pollsters can learn through trial-and-error the best ways of weighting their results to achieve accurate predictions.

So the fact that YouGov are good at predicting elections doesn't mean that their polls are any good at probing the nation's drinking habits, attitudes to the mentally ill, favorite vegetables, or whatever else. They could be totally wrong. Or they could be perfect. We don't know. It doesn't matter to the people who commission these surveys, of course, because it's publicity either way. It should matter to journalists, but it doesn't seem to.

Bottom line: if you want cheap media exposure, call YouGov. If you want serious news, don't. And if you want to know how journalism got into this sorry state, read Bad Science and Flat Earth News. Really. Bloggers like me are not going to shut about those books until everyone's read them at least five times.

[BPSDB]