Roll of statistics: Study design

A recent comment on the ESA (Ecological Society of America) list serve caught my attentions.

While I agree that too much emphasis is placed on statistics, phrases like the one below alarm me. I recall a talk by a distinguished professor in which he had his students review 400 papers and concluded that only two were scientifically correct. My conclusion was that 99.5% of scientists disagreed with his idea of how to do science.

I think that it is quite safe to say that scientists need better statistical training. Do they need to be trained statisticians? No. At the beginning of every study the students or PI or both should consult a statistician. Why? There are two reasons for this. Firstly the statistician can help to make sure that the study is appropriately designed so that they are indeed testing what they want to test. Secondly, the statistician can also advise them about what the best way to analyse the data are.

I remember hearing a story from a statistician at the university where I did my PhD. A MSc student in the Physiotherapy department wanted to see how balance was affect in amputees where the leg had been removed above the knee or below the knee. Towards this goal, they got in amputees and tested their balance and stability in different postures. If I recall correctly (the conversation occurred on a Friday evening at the Staff club), they’d obtained a hundred or so different measurements and then went to consult a statistician to find out how best to analyse their data. Unfortunately, while they had a lot of data, their actual sample size was too small to do anything with. They’d used two amputees, one who’d lost the leg above the knee, and one who’d lost leg below the knee. While their study should adequately demonstrate any differences in balance between the two amputees, trying to generalise their results would be extremely foolhardy.

So rather than use two individuals and take large numbers of measurements from them (pseudo-replication eg Hurlbert), what they should have done, would have been to have large numbers of individuals, and then take measurements from them since there is likely to be variation between individuals within the same category.

So do I agree with the above statement that there is too much empahisis on stats? If it's a well designed study then the stats become a side issue. On the other hand I hate reading papers and wondering why the hell people have chosen (in my oppinion anyway) an inappropriate means to analyse the date.

Hurlbert, S. H. 1984. Pseudoreplication and the design of ecological field experiments. Ecological Monographs 54:187-211.