If you hold a ruler against the image above, every line appears straight. Yet everyone perceives a tilt. This does not arise from a single tile, but from the arrangement as a whole. A larger wall with more tiles does not make the lines any straighter, and anyone aware of the optical illusion will still see it.
This is how a systematic error, known in research as bias, behaves. Unlike chance, it does not even out with more data. It is inherent in the design of a study, and every single measurement within it may be correct.
Added to this is a language that is far removed from the consulting room and from everyday life. A study does not ask: “Does this help?“, it asks: “Is the difference between the groups so great that chance becomes a highly unlikely explanation?“. The treatment is pitted against chance, and it is deemed effective if chance loses on the numbers.
The result is presented as an estimate with a range within which the true value is likely to lie, and is often phrased in double negative: “not significant, different from zero“. You have to mentally “translate“ such sentences back into everyday language. In doing so, it is easy to lose sight of the fact that ‘not proven’ means something different from “does not work“.
This issue is particularly acute for treatments tailored to the individual.
The standard clinical trial is designed to address a different research question: a drug, a diagnosis or a measured value is assessed at a specified point in time; subsequently, the two groups are compared on average. This approach works very well for a blood pressure-lowering drug. However, a treatment that responds to the individual does not fit easily into this framework in several respects:
- The group is formed on the basis of the diagnosis, whilst treatment is determined by the patient’s response. Three children with middle ear infection may, for example, be running a high fever and restless in one instance, pale and clingy in another, and listless and apathetic in yet another. They have the same diagnosis and require three different treatments, but are placed in the same group for the purposes of the study.
- Whether the treatment was appropriately chosen in each individual case is not recorded. Good and poor decisions are treated equally and averaged together.
- Treatment changes as the condition progresses. It is precisely these changes that provide information about what has happened. However, the analysis only considers which group a person belonged to and whether they achieved the pre-defined target – the so-called endpoint – such as a change in a measured value or a result in a questionnaire that differs significantly from the baseline values.
- People respond at different times. If one person improves after two weeks and another after eight, a measurement taken after four weeks will show only a non-representative aspect of both.
- The clinician is part of the treatment. A detailed initial consultation is both an examination and an intervention. It can neither be concealed nor factored out of the result.
- The differences between patients are often greater than the difference the study is seeking to identify. With small groups, randomisation does not reliably balance out such differences.
All these factors can then skew the result in the same direction: towards the conclusion that ‘there is no difference between the comparison groups’. This is not because anyone has done a poor job. The study design then dilutes the very effect it is seeking to detect. If, for example, five out of twenty patients benefit significantly and fifteen do not, the average shows only a ‘slight improvement’, and the study then concludes that the result is ‘not significant’. This is usually interpreted as ‘does not work’. However, it may also mean that the treatment helps a few people significantly and the mean does not reflect this, or that the study was unable to detect the effect at all – perhaps because measurements were taken at the wrong time or because the actual changes were not measured.
This is not a problem specific to any one form of treatment. It affects many areas, including physiotherapy, psychotherapy and general practice consultations – in other words, anywhere where a personalised response is given to the individual. Such treatments can be evaluated, but only using a study design that is appropriate for them.
We shall discuss here what such a framework might look like and what questions one should ask of a study before reading its results, particularly when it comes to homeopathy. We invite you to contribute your thoughts and voice any objections.