For many years this blog has had a focus on neuroscience, synaesthesia and autism, and I know that many of my readers have an interest in the autistic spectrum, for whatever reason. For those wishing to know more about the complex area of autism and Asperger syndrome the sensible advice has always been to look to evidence in the form of studies and reviews of research published in reputable scientific and medical journals. This is sound advice, but much too reassuring. Even the most supposedly sound sources of information such as university professors, published studies and science journals should be regarded with caution. Nothing and no one can be completely trusted, because the game of science is full of conflicting interests. This is a troubling fact that the popularizers of science and the cheer-squad for the scientific/rational world view happily overlook or dismiss as unimportant (I'm looking at you, Dr Karl).
Please read the recent news article linked to below and consider how easy it apparently was for academics in an Australian university (Prof. Bruce Murdoch, Caroline Barwood and M. L. Ng) to get a study published in an international science journal despite the fact that the study was apparently never conducted, and also please consider how they apparently managed to get their hands on a $20,000 grant for that phantom research study. Please also consider that the type of neuro-therapy that the dodgy boffins in the article were supposed to have administered in their study, repetitive transcranial magnetic stimulation, is one of the biggest new fads in autism cures, and has been championed by the celebrity "aspergian" autobiographer John Elder Jobison. One Australian university has been conducting a research program in repetitive transcranial magnetic stimulation as a treatment for autism for many years (I wrote about and criticised those researchers a number of years ago at this blog). Please also consider the fact that autism research attracts copius funding from government and charity bodies, and therefore could well be a field of research that would attract researchers who are more interested in dollars than knowledge. What percentage of autism research is a complete crock of shit? We can only speculate, but there's a pretty strong smell, and it isn't a pleasant one.
University of Queensland investigates paper by ex-staff member citing 'no evidence' of research.
by Melinda Howells
ABC News.
4 Sep 2013.http://www.abc.net.au/news/2013-09-03/uq-investigates-study-that-might-never-have-happened/4932648
The paper itself:
http://onlinelibrary.wiley.com/doi/10.1111/j.1468-1331.2011.03524.x/abstract
Retraction Watch on the paper:
http://retractionwatch.wordpress.com/category/by-journal/european-journal-of-neurology/
More media coverage:
http://www.brisbanetimes.com.au/queensland/university-of-queensland-professor-let-everyone-down-20130905-2t7un.html
It's worrying how weak the peer review system appears to be.
ReplyDeleteI made a quick search for similar papers with the same authors, and found one called "Improved language performance subsequent to low-frequency rTMS in patients with chronic non-fluent aphasia post-stroke" (http://www.researchgate.net/publication/49664846_Improved_language_performance_subsequent_to_low-frequency_rTMS_in_patients_with_chronic_non-fluent_aphasia_post-stroke/file/d912f50b9384a1351d.pdf).
I gave it a quick read to see if there was anything fishy about it. Pretty soon there was an obvious error - the control group were described as "aged between 51 and 85 years", but the ages of the six people in the group are given on page 4 as 72, 51, 77, 66, 71 and 63. The obvious error here is that there is no 85 year old in the group.
This could, of course, be just a typo. But if it is a typo, it would at minimum suggest that this paper has not had any really careful peer review and scrutiny. This sort of glaring error is exactly the sort of thing a peer reviewer ought to pick up.
And how was the error made in the first place? It could be nothing more sinister than a typo. Or it could be a clue that there was no real data behind this experiment either, and that like the other one the results are just being made up.
Either way, given the authors can't get their basic facts straight, it suggests that this is another paper that is not to be trusted.
It staggers me just how easy it is to find basic errors in peer reviewed scientific papers - the quality standards sometimes seem to be at tabloid newspaper level. I'm thinking of starting a blog about it. If only I had the time.
Tomas
You are definitly onto something here, Tomas. We are the detectives of science! People like Retration Watch publish retractions (more than they ever imagined they would find), but it appears that it is up to bloggers and blog commenters to identify aspects of many published journal papers reporting studies that don't add up or that breach the established methods of science.
ReplyDeleteThere is definitely a mistake in the paper. Where is the placebo group subject aged 85, mentioned in the text? Not in Table 1 or it's continuation. Conceivably it could be a typo or some innocent mistake. A typo seems unlikey because in the sham group I found no subject with an age starting with the digit 8 or ending with the digit 5.
Something is amiss, and in light of recent events this paper and study needs to be investigated. It has 8 authors, two of them from a hospital dept of neurology and one from a hospital dept of medical imaging. If the study was a complete fabrication I've got to wonder how these authors were connected to the study. My bet is that it's sloppy but not a complete fictioin. The other study which is under serious question had only 3 authors, none of them from a hospital, and maybe that is an important point. Should science journal reviewers be wary of papers that have a less than expected number of authors? Why aren't they doing this already?
On reflection, it's worth noting that strictly, the statement that they are "between 51 and 85" years is correct, as they all fall within this age range, even if the oldest is 77 rather than 85.
ReplyDeleteHowever, the figure given for the standard deviation, of 13.11 years, is unambiguously just plain wrong; the standard deviation calculated from the data is 9.09.
I agree with your view that it's more likely to be just sloppy analysis than plain fiction. But it's looking very sloppy - how can we trust their detailed statistical analysis when they can't correctly calculate the most basic statistical amounts like a standard deviation?
Tomas
Sloppiness shouldn't be seen in paper in science journals. Sloppiness is reason enough to question whether this paper should be published in it's present form.
ReplyDeleteDo you think it looks like the authors have pulled an outlier out of the data set during their analysis, and they've forgotten to re-do the stats?
The standard deviation of 13 is very large. It's possible that it's explained by excluded outliers as you suggest, but looking at the numbers I think it's unlikely.
ReplyDeleteFor example, I calculate that if they had excluded two outliers, one age 47 and one age 87, and then forgotten to rerun the numbers, that would roughly give the right mean and standard deviation. But those are both well outside the range of the other ages, so looks unlikely to me.
Tomas