Methodologies tell people how to do things: they describe steps in a process. A useful rule of thumb is that gotchas for a methodology cluster around the steps the methodologists aren’t interested in. The critical rationalists aren’t interested in experiments. They are Big Design Up Front, Command and Control people, and their methodology suffers because of it.
My father (RIP) built houses. When he first went independent, people wanting a house would bring him an architect’s plans/blueprint, ask him how much building a house to match it would cost, then contract him to build it.
He hated architect’s plans, for two reasons that I remember:
-
Architects would include a feature designed in a way that was hard to build. That didn’t matter to Dad’s bottom line – he was good at estimating cost He was once off by one dollar in his bid to build a house. He got lucky, admittedly, on the things that are hard to predict, but an error of one part in maybe 40,000 (given prices of the time) is not bad. – but it offended him that his clients had to pay more for a feature that could be done just as well but more cheaply.
-
Architects tended to be detached from their client’s reality. I remember one time when Dad was building a house for a couple where the husband had a job that left him dirty at the end of the day, the way construction does. Such a person showers after getting home from work. A user-friendly house plan provides a “straight shot” from the garage to a bathroom with a shower. The architect’s design had the husband tracking dirt all through the house to get to the shower.
When Dad was successful enough, he simply stopped working with architect’s plans. If you wanted him to build you a house, you had to agree to collaborate with him as he drew the plans.
From this and my own later experiences in software, I’ve come to believe that abstract understanding fails unless combined with nitty-gritty implementation details. That applies to both the design of objects (like houses) and also design of methodologies.
With that context in mind, let’s talk about an experiment.
Solar neutrinos
The Homestake experiment tested a theory of how the sun works. It found fewer neutrinos than predicted. An adjustment or addition was made to a standard theory, and science marched on.
I could cast this as an example of how critical rationalists say science should work. At an abstract level, it is. At a concrete level, things get more interesting. I’m largely drawing on the description of the experiment in Dudley Shapere, “The concept of observation in philosophy and science,” Philosophy of Science, 49 (1982) pp. 485-525. Plus Wikipedia.
Although the scientists talked casually about “seeing neutrinos,” in the actual experiment what was observed was not neutrinos, but the number of clicks made by a glorified Geiger counter exposed to some very mildly radioactive argon gas. There’s a long chain of justification between “we counted X clicks in time Y” to “the sun produces neutrinos at such-and-so a rate.” Here are some of the links in the chain.
The argon gas fed to the detector was made by combining a very small amount of an unstable isotope of argon with a vastly larger number of off-the-shelf argon atoms. By “very small amount,” I mean that the experimental design estimated (based on theory) that a multi-day run of the experiment would produce around 200 atoms of radioactive argon. That number of atoms is not exactly easy to work with, hence the combination with enough argon that you can take advantage of known methods of working with gases.
Where did the radioactive argon atoms come from? From the highly unlikely but theoretically possible interaction of a neutrino and one isotope of chlorine. So unlikely is that interaction that you need a vast, vast, vast amount of chlorine to produce an estimated six atoms of argon per day. Fortunately, cleaning fluid (perchloroethylene) is mostly chlorine, and apparently it’s cheap enough that a mere 400,000 liters of it was estimated to contain enough of the right chlorine isotope.
Another problem was that neutrinos aren’t the only things that can convert chlorine to argon. Muons – found in cosmic rays – can too, but burying the fluid chamber 1,478 meters below the earth should have ensured that only a trivial number of muons reached it.
Alas, muons can also be created by the interaction of sulphur with fast neutrons (from radioactive decay of uranium in rock surrounding the fluid chamber), so they had to make sure the cleaning fluid wasn’t much contaminated with sulphur (less than one part per million).
Solving these and a few more problems led to an estimated one “spurious” argon atom every two days – 1/36th the expected sun-caused rate. They were very careful to avoid overcounting. But it turns out that overcounting wasn’t a problem because…
After all that careful setup (and more), only about 1/3 of the neutrinos predicted by theory were detected. There ensued debate about how to resolve what came to be called the solar neutrino problem.
One possibility was that the theory being tested – the standard solar model – had been refuted. But not so fast! That model is based on a simple formula for particle interactions:
That says that a star works by smashing four hydrogen atoms together to produce a helium atom, two electrons, and two neutrinos. But there’s more to the process than a pretty formula. There are three different “pathways” from the starting state to the finished state, and how often each pathway is taken depends on the density and (especially) the temperature at the core of the sun.
Is it the formula that’s been refuted? Or hypotheses about the environment at the sun’s core?
Alternately, what about other parts of particle physics? The estimates for how often neutrinos interact with chlorine are derived from a theory of the weak interaction. Maybe what’s at fault is a misunderstanding of the weak interaction? (So that what was refuted was not what was being consciously tested, but a “nearby” theory?)
Or maybe neutrinos can decay and so only a third of them make it to earth?
Or what about chemistry? The argon atoms were collected by bubbling helium through the liquid in the chamber, then separating the argon from the helium with a charcoal trap. Maybe something went wrong there? For example, argon was chosen as the target atom because it’s a noble gas, one reluctant to participate in chemical reactions. But perhaps a new argon atom is born ionized and so can “stick” to a perchloroethylene molecule, in which case the helium trick wouldn’t work for that atom. Maybe 2/3ds of the argon atoms were missed for that reason? Or (in an alternate theory) maybe the new argon atoms are unavailable to the helium because they’re structurally “caged” in the molecules in which they were just created?
The anomalous results were detected in the early 1970s. Time passed, with people working on explanations like the above. By the time of Shapere’s paper – a decade later – matters were unresolved: Shapere 1982, p. 499, footnote 7.
The most serious possibility lies in the hypothesis of neutrino oscillations, which is being actively considered by physicists on other grounds. However, there remain grave doubts as to whether that hypothesis could really reduce the predicted neutrino flux to the observed level.
Neutrino oscillation holds (roughly) that neutrinos oscillate among three states as they move through space. Only one kind is energetic enough to have been detected by the Homestake experiment. The oscillation idea actually predates the Homestake experiment by a decade, but was untestable at that time. Homestake offered some weak confirmation for it (since the observed shortfall of 2/3 is consistent with it.)
It was not until 1984 that Herb Chen pointed out that heavy water could be used to address the question. After Atomic Energy of Canada Limited promised to lend experimenters CA$330,000,000 of heavy water and the owners of a really deep mine offered to lease part of it cheaply, the Sudbury Neutrino Observatory was set up. Per Wikipedia, in a mere sixteen years, Sudbury “provided clear evidence” that, um, failed to refute the conjecture of neutrino oscillation. Neutrino oscillation implies that neutrinos have mass, so this lack-of-refutation served as a (partial) refutation of the Standard Model of particle physics.
Evidence and methodology
This is roughly critical rationalism. Predictions were made. When they failed, theories were adjusted, and novel predictions resulted. More weight was given to confirmations of theory than Popper would have liked (though Lakatos would have likely approved of them as dramatic enough – akin to the return of Halley’s comet).
But notice the word “experiment” doesn’t appear in the previous paragraph. In both discussions of their methodology and case studies, the critical rationalists treat experiments as quick and conclusions as obvious. They don’t describe thirty-year sagas of people trying to figure out what a result means.
I shouldn’t be (too) unfair. Popper, in particular, devotes some time to things like the Duhem-Quine thesis. That’s the argument that any experiment that tests a theoretical prediction relies on a whole swath of “theories of the instrument.” It might be those theories that have been refuted, not the theory under test. There’s no principled, or rational, way to finger the culprit.
But having raised the issue, Popper drops it.
First, he appeals to “basic statements.” The Duhem-Quine thesis allows any attack you make on my theory of the instrument to be countered by my claim that your attack relies on a theory that itself is wrong. There is a recursive descent of theory-based claims and counter claims. Popper suggests that such claims can bottom out with basic statements: “Simple descriptive statements, describing easily observable states of physical bodies.“ Karl Popper, Conjectures and Refutations: the Growth of Scientific Knowledge) (5/e), 1989, p. 267.
But there’s a big gap between descriptions of the Homestake experiment’s tower of supporting claims and basic statements like that a counter clicked n times. No real argument conducted by actual humans will descend to basic statements, so this is a dodge that explains away a real problem rather than solving it.
Popper’s second gambit is what he calls “background knowledge”: Conjectures and Refutations, p. 238.
Almost all of the vast amount of background knowledge which we constantly use in any informal discussion will, for practical reasons, necessarily remain unquestioned; and the misguided attempt to question it all – that is to say, to start from scratch – can easily lead to the breakdown of critical debate. (his emphasis)
This is puzzling. Scientists are supposed to simultaneously relentlessly probe theories while at the same time just accepting the previously-established experimental background knowledge?
I don’t think you should adopt the mantle of “critical rationalism” if you choose what things it is rational to criticize. Is it perhaps that theories are objects of fascination to Popper whereas interpretation of experimental results is grubby and boring? Popper’s variant of Platonism describes three (metaphorical?) worlds. Per Wikipedia, World 1 is the physical world-states and world-processes that science studies – the physical world. World 2 is the realm of mental states and processes, such as sensations and thoughts. These arise only through biological processes. World 3 is ‘the products of thought’ as they float free of particular bodies – the world of culture or ideas. Critical rationalism is intended to regulate which ideas make it to World 3 by regulating the processes taking place in the bodies and minds of World 2. World 1 (the world in which neutrinos exist) is beneath Popper (metaphorically).
(OK, maybe I’m not so intent on being fair.)
Lakatos is even more dismissive of experiment. Whereas Popper seems to grant experimenters the right to interpret their results and so feed information to the theorist, Lakatos disdains experimenter judgment. I’ve already noted his claim (to Kuhn’s dismay) that the Balmer spectrum of hydrogen was entirely superfluous to the development of the Bohr atom: Lakatos claims the “positive [theoretical] heuristic” of the great particle theorists would have taken them to the same theories without the experimental evidence.
Moreover, Lakatos thinks that the great theorist ought to overrule the experimenter by insisting which theories of the instrument the latter is allowed: Criticism and the Growth of Knowledge, Lakatos & Musgrave (eds.) (1970) (full text), p. 130, footnote 5.
A classical example of this pattern is Newton’s relation to Flamsteed, the first Astronomer Royal. For instance, Newton visited Flamsteed on 1 September 1694, when working full time on his lunar theory; told him to reinterpret some of his data since they contradicted his [Newton’s] own theory; and he explained to him exactly how to do it. Flamsteed obeyed Newton and wrote to him on 7 October: ‘Since you went home, I examined the observations I employed for determining the greatest equations of the earth’s orbit, and considering the moon’s places at the times . . . ’ I find that (if, as you intimate, the earth inclines on that side [that] the moon then is) you may abate abt 20” from it.. .’ Thus Newton constantly criticized and corrected Flamsteed’s observational theories. Newton taught Flamsteed, for instance, a better theory of the refractive power of the atmosphere; Flamsteed accepted this and corrected his original ‘data’. One can understand the constant humiliation and slowly increasing fury of this great observer, having his data criticized and improved by a man who, on his own confession, made no observations himself: it was this feeling — I suspect — which led finally to a vicious personal controversy. (Italics Lakatos, I think; bolding mine.)
One imagines Lakatos swooping into Homestake like the XKCD physicist:
He would probably swoop out again before coming to the necessary later realization:
The consequences
The critical rationalists treat experiments as if they provide little more than one bit of information: confirmed or refuted. Popper allows that a refutation can point to which of the universal claims in the theory need revision or replacement, but also recommends that the theorist avoid confronting the refutation too directly.
I failed to make a note of where I saw that advice, so I can’t provide a page number. I think it was in Conjectures and Refutations. I suppose he doesn’t want theorists tempted into a theory revision that just explains away a refutation without producing bold new predictions.
These rationalists do not allow for more information to flow from experiment to theory. For example, consider this statement from the Wikipedia article on neutrino oscillation:
Neutrino oscillation is of great theoretical and experimental interest, as the precise [measured] properties of the process can shed light on several properties of the neutrino.
That’s not a thing in the critical rationalist world. The idea of collecting a lot of information as a way to inform theoretical creativity goes unmentioned. Popper cites Einstein approvingly: Popper, The Logic of Scientific Discovery, (translation by the author of Logik der Forschung (1935), 1959), pp. 8-9 in the 2002 printing.
There is no logical path leading to [the highly universal laws of science]. They can only be reached by intuition, based upon something like an intellectual love of the objects of experience.
… but does not suggest that such intuition is obtained by groveling through either observed or experimental facts. I imagine he would disagree in principle with this comment on Darwin’s eight-year obsession with dissecting barnacles:
Darwin’s study of cirripedes, far from being merely a dry, taxonomic exercise, was a highly theoretical work that addressed several problems at the forefront of contemporary natural history.
Darwin mixed observation and theory in a way that reminds me of how test-driven design is simultaneously design, implementation, and testing, so tightly connected that it’s absurd to think of them as separate “phases.”
Such a mixing of theorizing and experiment was as foreign to the critical rationalists as mixing designing and building would have been to the architects my dad avoided. Theorists predict; experimenters check; and if the twain ever should meet, that’s a departure from methodological rationality.
It’s notable how much the critical rationalists point to the Newton of the Principia, but brush aside the Newton of the Opticks. That latter work…
… is largely a record of experiments and the deductions made from them, covering a wide range of topics in what was later to be known as physical optics.
Unlike the Principia, […] the stated propositions are demonstrated by means of specific, carefully described experiments. The first sentence of Book I declares “My Design in this Book is not to explain the Properties of Light by Hypotheses, but to propose and prove them by Reason and Experiments.” (my bolding)
So when Newton was schooling Flamsteed on diffraction, he was doing so based on conclusions from his own experiments, which fact Lakatos characteristically doesn’t mention. Great theorist Newton fussing with prisms in Cambridge is as uninteresting as great theorist Darwin messing with barnacles.
That’s a core problem with critical rationalism: science relies on experiment. Everyone knows that. But the critical rationalists fall into the old, old trap of a binary hierarchy between abstraction and concreteness, theory and practice, architectural blueprints and the physical reality of houses and their occupants. That makes their advice fit badly with actual science and, I claim, would damage actual science if their methodology were followed.
Fortunately, as Peter Medewar wrote:
Most scientists receive no tutoring in scientific method, but those who have been instructed perform no better as scientists than those who have not. Of what other branch of learning can it be said that it gives its proficients no advantage; that it need not be taught or, if taught, need not be learned?
Actual scientists ignore what is still considered “the scientific method,” just as my dad ignored architects when their plans told him to do something stupid, just as programmers and software testers “in the trenches” ignored the methodologists as much as they could get away with.
I’m not saying that there should be no methodology of science, but rather that it should be more connected to the reality of the scientific practices that have been so surprisingly effective.
Marxism, again
If scientists were to really believe critical rationalism, they’d likely produce conclusions – frankly, embarrassing conclusions – like those I’ve documented for Popper and Lakatos’s proclamations about Marxism (here and here and here).
If you don’t believe observations (direct or indirect) of reality matter much, you won’t dig into the question of whether what you’ve gathered from your preferred sources – your “background knowledge” – actually happened. You become an ungrounded, unreliable narrator. That’s bad.
(The critical rationalist’s impoverished view of what theories actually are is more of a problem, but that’s a tale for another time.)