Posted on March 20, 2015

In 2011, with Ireland’s economy in the depths of recession and a general election imminent, several prominent journalists and commentators seemed to be about to launch bids for election to the national parliament as a new party called Democracy Now. They had, the argument went, been pointing out precisely what politicians had been doing wrong for many years and now was the time for them to cross the line and actually try it out for themselves. In the end, they decided against it and the politicians continue to get it wrong while they continue to point. There’s a similar line between statistical analysis that leads to policy recommendations and the implementation of those policies, and a decision to make about how close to it you want to get.

The census of everything

A line is drawn around the potential impact of any analysis at the design stage and with considerations about return on the research investment. To take an extreme example, a population census is a perfect opportunity to get nationally representative data on every topic you can imagine: standardised psychological assessment tools, income, education, and attitudes to advertising or to zoo animals. Of course, that would make a lot of statisticians very happy but would make for a census form the size of the Encyclopaedia Britannica that would take householders weeks to complete. Some collaboration with others is necessary to decide the bounds of the survey and which questions will be most useful.

Because researchers spend so much time on one topic, so much energy convincing themselves and then everyone else that what they’re doing is the most important thing in the world, there’s a risk that they’ll prioritise their research area over any potential impact. Even when they’re making a case for the relevance of their work, look closely and it might just be that they really want to keep working in the area. There’s also something cosy and familiar about staying inside the lines so they might be reluctant to look in detail at how to implement what they recommend.

There are two options: put your perfect research out there and then scoff at policy-makers’ stupidity for not doing what is so obviously right; or work with them, learn their language, and listen to their ideas from research question to final report. What you’re trying to avoid is fighting with the policy-makers, because if you’re fighting with them they’re not going to implement your world-changing ideas. It would be naïve to think that policy uses the best available evidence based on sound statistical analysis so part of the role of a statistician is to explain what the numbers actually mean, but there is also likely to be some tension.

Policies, you see, tend to be about tackling something, so if your analysis is in any way supportive of the status quo, don’t expect to get too far with the more righteous of policifiers. This is one of the issues underlying publication bias, that is, the tendency to report analysis that shows significant differences and suppress, wittingly or unwittingly, non-significant differences. People who dare to present conference papers on interventions that showed no significant impact have been described as “brave”. Sometimes it’s necessary to stick to your numbers and say “We can’t say for sure” or even “It doesn’t work”.

<<<<<<< Read On >>>>>> http://bit.ly/1ExfYTs 

You may leave the list at any time by sending the command

SIGNOFF allstat

to [log in to unmask], leaving the subject line blank.