Or, why reporters get a bad rap.
Maths is hard. Well, it’s not really, but many people, for whatever reason, decide that it’s too hard for them. Unfortunately, sometimes they go on to use that lack of knowledge to do damage to not only themselves but the industry in general. Enter Kotaku, stage left.
About a week ago, Kotaku ran a poll to see how many of their readers owned a high-definition TV and, for those that didn’t, asked why the respondent hadn’t upgraded yet.
There’s a variety of issues with this survey. It can’t be generalised to anyone other than Kotaku readers because it’s biased by self-selection and there’s no way of knowing the socio-demographic characteristics of those that responded. The numbers don’t match, given that 2,929 respondents said that they hadn’t upgraded to HD but 4,614 respondents gave their reasons why they hadn’t upgraded (a response rate almost twice as high as it should have been). Even worse, no attempt was made to cross-reference the two surveys by IP or session to provide some level of error control. The available responses were limited and the language introduced bias – the Wii was singled out as the single console-based excuse for sticking with SD (despite the PS2 still having a significantly greater footprint). On top of that, no allowance was given for the market that owns a large standard-definition panel, a product which made a significant proportion of sales in international markets for the first half of the last decade.
In short, the survey was fundamentally flawed to the point where without careful wording, the odds of misleading the reader were extremely high. However, that’s not the problem.
The problem isn’t that journalists aren’t statisticians; why should they be? That’s like complaining that the designers behind World of Warcraft aren’t Nobel Memorial prizewinners in economics – after all, the game deals with macroeconomic systems, doesn’t it? While it’d be great for everyone to be experts in everything, it’s just not realistic. However, what we should expect from those who have a certain level of credibility is an acceptable standard of insight and quality control. I sincerely doubt that Crecente or Totilo would feel comfortable letting someone with little to no editorial experience choose future writers for Kotaku. Assuming that’s true, why should it be different for a profession in which they haven’t been trained?
It may look like I’m picking on Plunkett here – I want to make it clear that I’m not. He does good work, he had a good idea, and he ran with it. The problem is that this is an endemic issue, not only in game-related journalism but generally; it’s also one that’s incrementally damaging the reputation and sustainability of the profession. People read news because they trust it. They read it because it’s entertaining. They read it because more often than not, it’s right. Every time an article goes up that generates hits but is rapidly debunked as misinformation, the long-run credibility of the publication and industry in general is damaged.
The single biggest point of differentiation between a commercial news entity and, say, this one, is the level of financial commitment behind it; the rumours of hookers and blow behind the scenes here, despite being somewhat cool, are vastly overstated. We don’t have the money or resources where they do and yet despite this, they ran an editorial article that they didn’t fact-check with someone who had professional experience. It’s not like statisticians are hard to find, either – after graduating, we tend to cluster. See what I did there? Cluster? Anyway …
Every time something like this runs, the point of differentiation between the commercial publication and the volunteer writer narrows. We write sporadically and we don’t have their finances but we’re competing for the same set of eyeballs and we’re legion; if the quality of output on a commercial publication is no better researched or considered than your average blogger with a SurveyMonkey account, where’s the value proposition? Kotaku’s made some giant steps towards decent editorial content lately. Unfortunately, running something like this simply cheapens all the good work they’ve done so far. Maybe not to the majority of their existing audience, but definitely to anyone with a high-school level stats education. In an industry desperately struggling to survive and remain financially viable, you quite simply can’t do this. Ever. The costs of failure are too high – have a look at the gradually decaying empire of the mainstream press sometime.
A lawyer gives bad advice, they get sued. A doctor misdiagnoses and they end up with a malpractice suit. A project manager mismanages and they get sacked. But a journalist? They write a badly researched article and while nothing happens to them immediately, over time they kill the industry by a thousand cuts.
So what’s the answer? Basic journalism 101 – if you’re going to run an editorial on a specialist topic, make sure you either understand the content or talk to someone who does. Open response surveys can still be extremely useful, it’s just important to understand their limitations, design them correctly, and accurately present their results. And, whatever you do, don’t imply that because there’s a high level of response, the results must be reasonable. Nothing gets a statistician more riled than a bad application of sampling theory. Seriously.
You can count on it.
Related posts:
- On inclusivity
- This is Week – Abused Feelings in 3D Edition
- This is Week – Compassionate Gamers Edition
Tags: Kotaku, maths, samping, Statistics, surveys
