I caught Harry Potter and the Half Blood Prince when it showed on TV and I felt rather envious of Harry Potter. My envy was because Mr Potter was not having a problem that I routinely have (and not because of the $15 billion brand, place at Hogwarts and ability to defeat evil).
He is given a second hand text book to use in his potions class. The level of detail provided in the printed text is woefully inadequate making brewing your potion almost impossible, his classmates struggle to brew their ‘draught of living death’ and all fail. Harry’s second hand book contains a number of annotations (written by the ‘Half-Blood Prince’) that provide the details, omitted from the printed text, required to brew the potions. Moving from Hogwarts to the Henry Wellcome Building life isn’t that different in the science laboratory.
Science papers supposedly contain a methods section, but few are detailed enough to actually follow. They often reference other papers (which in turn reference other papers), fail to include sufficient detail or contain details that are obviously wrong. For example a paper that I read recently (which I shall not name) describes the following method:
The technique used to acquire and analyze echocardiographic data in individuals has been described48. The mice were imaged under light sedation (1–1.5% isoflurane) and the data were analyzed as described49.
48. Schiller, N.B. et al. Recommendations for quantitation of the left ventricle by two dimensional echocardiography. American Society of Echocardiography Committee on Standards, Subcommittee on Quantitation of Two-Dimensional Echocardiograms. J. Am. Soc. Echocardiogr. 2, 358–367 (1989).
49. Haq, S. et al. Deletion of cytosolic phospholipase A2 promotes striated muscle growth. Nat. Med. 9, 944–951 (2003).
I would be surprised if there had been no developments in echocardiography in the 17 years between the publication of the paper (2006) and the referenced methods paper (1989). It is also fairly obviously that their method diverged from the referenced paper, I’m unable to read the full text of Schiller’s paper, but it appears that their paper refers to humans whereas the paper mentioned above relates to mice. This is not a unique example, but a problem that is endemic in the scientific literature.
Imagine the following conversation:
Friend A “I cooked this amazing curry last night, it was the best curry in the history of mankind. It tasted wonderful, smelt great and it cured all known diseases”
Friend B “Wow that sounds like the best thing ever, I need to try and cook this myself – can I have the recipe please?”
Friend A “Chicken (ASDA) and onion (Tesco) were cooked on a hob (BOSCH)”
The claims made about the curry are not that far off the claims made by some scientific papers (and even closer to some of the claims made by the press office) and while I may not be the greatest cook in the world I think even the most starred Michelin chef would struggle to replicate Friend A’s elixir curry. Yet we see papers with phrases such as “Protein expression levels…were quantified by determining band density with the MCID analysis application from Imaging Research” appearing all the time. For those unfamiliar with quantification of protein expression levels lets go back to Friend A and B again for an analogy:
Friend A “I have worked out a way of reducing the nations’ deficit without making any cuts”
Friend B “Wow how did you do that!? That is amazing!”
Friend A “I used Excel from Microsoft”
I think people would soon stop listening to friend A and his terrible ideas.
Now why is this a problem? Science papers should not be considered ‘authoritative’, the reason they are published is for other people to replicate, and verify, their results – so including detailed method sections would be beneficial. This becomes problematic when a company wants to develop a medicine to treat Alzheimer’s based upon friend A’s elixir curry, or, more likely, papers with titles along the lines of “modulation of protein X inhibits neurodegeneration and prolongs survival in a mouse model of Alzheimer’s”. While estimates vary, it is thought that about two thirds of these papers cannot be reproduced in these labs [1].
One suggestion that has been made is to contact the original authors for advice [2]. However, this has problems too. The corresponding author is usually the principle investigator who is probably more familiar with their office than their laboratory. With the poor job security associated with early stage researchers, the chances that the researcher who carried out the research still being in the laboratory is low… so is the chance that they are still working in research. Having to hire a private investigator (or having to have an intensive google session) each time you want to find out how someone cultured their cells is not ideal.
The written protocols held by laboratories can also be rather ropey. I’m sure nearly everyone who has worked in a laboratory has been given a supposedly current protocol that includes the use of equipment that has been broken for a number of years or wondered how a fellow lab member completes a two hour step in 30 minutes. Protocols move on and are often passed on Chinese whispers style from worker to worker with the written protocols left describing equipment that has been sitting in a state of disrepair for years. Hard copies with written annotations, like the Half-Blood Prince’s potions book, can be absolute gold dust. The annotations telling you incubate for 10 minutes not 2, change the rota speed, and removing antibiotics from the media can be the difference between a frustrating and a productive week. If a protocol is passed on it is unlikely to include these important details.
So what can be done? Ideally there would be a revolution in scientific publishing meaning that detailed protocols are included with papers along with all the data. Before that happens changes can be made by us. Ensuring that we all maintain accurate written copies of our protocols is the first step. The second is that we live at a time when it takes two seconds to share what we had for dinner, or what we did at the gym. There is no reason why we cannot share our detailed, and accurate, protocols online.
In conclusion if anyone has the Half Blood Prince’s book of pharmacology protocols could you lend it to me please?
By Harry Holkham
[1] http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html
[2] http://www.nature.com/news/reproducibility-the-risks-of-the-replication-drive-1.14184
Subscribe to Harry's posts
Recent Comments