The history of modern science includes an ongoing struggle to get the art out of science. It would appear that we must do so on pain of foregoing science’s claim to objectivity. On Worrall’s report and analysis, the “Evidence Based Medicine” (EBM) movement is another chapter in such efforts, and the moral that I draw from his analysis, a moral that Worrall does not himself draw, is that, once again, we are pressed to face the circumstance that science is, among other things, a craft, a spectacularly well developed branch of the “art of knowing”. I would be most interested to know whether Worrall would or would not agree with this (meta) assessment.

On the one hand, Worrell deploys the rhetoric embodied in such phrases as “the scientific method” , “[the] scientific evidential approach”, and “universal general principles of the logic of evidence”. But on the other, his analysis would appear to illustrate the implausibility of their being any universally applicable principles and to illustrate the fact that there is no such thing as “the scientific method”, but rather there are many scientific methods (plural!), ones that already exist and continually undergo adjustment and refinement, and new ones always under construction.

Worrall examines the view that in medicine “randomized controlled trials” (RCTs) provide the “gold standard” of evidence in medicine – a form of evidence that, if not the only sound form of evidence, at least one that trumps all others; and he does a delicious job in exploding the arguments for this view and in showing, by wonderful examples, how very implausible it is. He does not argue that randomization is never useful, but, rather, that it is one evidential tool among many and that one to which we must apply “scientific commonsense” in sorting out its role in a complex evidential landscape. His avowed aim, however, is not just to criticize, but to “indicate a number of areas where philosophers of science can contribute to a proper implementation of exactly that scientific-evidential approach”. These areas include questions about how evidence from different randomized trials are to be amalgamated, more generally, how different types of evidence are to combined, whether, and if so how, does randomization help to control bias, what work can randomization do in controlling for known and unknown confounding variables, and more generally, what the tool of randomization in fact can accomplish.

Another claim that Worrall does not make, but with which I am extremely confident he would agree: In application to medicine these are all medical questions well informed answers to which will depend on deploying medical expertise. At the same time, Worrall demonstrates, by numerous illustrations, that philosophers of science can contribute valuable expertise in examining such issues, for example by deploying our skills in argument analysis, and in bringing to bear more general considerations, for example by deploying more general Bayesian critiques of standard statistical methods.

Worrall raises a further question that is not more narrowly medical. Why the usefulness or randomized trials in medicine and many other subjects, but almost never in physics? And suggesting a possible answer to this question provides a useful restatement and generalization of another of Worrall’s questions.

Here is my suggestion: In physics one can often very finely control all the relevant variables. One can arrange an experimental situation in which all but an electrostatic force are negligibly small. One can arrange experimental conditions under which light can be treated as rays, a fluid as an incompressible continuous medium. There are also clear cases in medicine: As Worrall points out, we don’t need RCTs to ascertain whether appendectomy is an effective therapy in the case of acute appendicitis. But in medicine we often want to intervene in cases in which we can’t begin to control all the relevant variables. To be sure, such cases frequently arise in manipulating the physical world, for example, in testing the effectiveness of a fuel additive in increasing mileage in your car. There are a great many potentially confounding variables: temperature, humidity, brand of gas, road surface and other driving conditions, to say nothing about possible biases from the driving behavior of the driver. Here an RCT might make good sense. But note also that it is in precisely such cases that we no longer talk about “physics”, but rather “applied science” or “engineering”

These considerations suggest that the appropriateness of RCTs is restricted to subjects, such as psychology, agriculture and many others as well as medicine, in which we are dealing with enormously complicated situations that are hopelessly beyond any possibility of simplification or exact analysis. One of the ways that we can succeed in manipulating our environment even when there are many relevant variables that we cannot control is to identify relatively stable environments. As long as the other relevant variables do not vary too much, we may find that we can influence a situation my manipulating one of the few relevant variables that we do know and which we can adjust. But when we step out of the stable environment, our controlled variable may well no long bring about the desired effect: “Normally”, striking a match will result in its lighting. But not when here is insufficient oxygen.

I suspect that talk about “confounding variables” presupposes such a notion of a relevantly stable environment. In an otherwise stable environment there may be a number of non-constant, potentially relevant variables, the one under test, and other “confounding” variables, such as patient care. When we satisfy ourselves, with RCT or other methods, that we have identified an effective treatment variable, we will have done so only within the context of a relatively stable environment. Change the environment, and the erstwhile effective treatment variable may cease to be effective. To provide an oversimplified example. In a context in which there is plenty of sunshine or other sources of vitamin D, calcium supplements may be found to be an effective treatment of bone density loss in the elderly. Change the environment in which there is insufficient vitamin D and the effectiveness of the treatment evaporates.

Here is another way to put the issue. In practice, no treatment is effective in all environments. Once potentially confounding variables have been eliminated for a presumed environment, there is still the question of how wide the environment is in which the treatment is effective or in which in environments it is effective.

Perhaps this is just another way of putting an issue that Worrall does discuss: A test that shows a treatment to be effective under test conditions is said to have internal validity. But there is still the question of the test’s external validity, that is the question of to what other target populations the results of the test can be generalized. But putting the issue in terms of stable environments, or in terms of “standing” and “variable’ conditions, both generalizes the issue and indicates another way in which problems of analysis and control of exceedingly complex causal networks defies any humanly accessible closed solution.