Reproducible experiments (again)

Last year I discussed a psychologist's essay on "the emptiness of failed replications". I'm returning to the status of experiment replication because a few days ago Science published a paper of the Open Science Collaboration, which was only able to confirm less than half of the 100 psychological studies it replicated. The study is very interesting in itself, but I'll only comment here on the reaction of psychology professor Lisa Feldman Barrett in a New York Times op-ed. She makes the point that attempting to replicate an experiment in different conditions can lead to different results and, by that fact, point to the importance of previously neglected parameters: context is important.

She illustrates her position by three examples, one of them taken from physics :

[...] when physicists discovered that subatomic particles didn’t obey Newton’s laws of motion, they didn’t cry out that Newton’s laws had “failed to replicate.”

Prof. Feldman Barrett confuses here the epistemological function of experiment and theory. A physical theory cannot be replicated, but it can be tested using (hopefully, reproducible) experiments. Obviously, performing an experiment with elementary particles cannot be seen as replicating one that uses macroscopic objects.

There is, however, a more alarming problem with her position, summarized by the phrase :

Much of science still assumes that phenomena can be explained with universal laws and therefore context should not matter.

There are very few completely general natural laws. Scientific results are "universal" precisely in the sense that the necessary context should be fully specified : A occurs every time conditions B, C, and D are fulfilled, irrespective of parameters E, F and G that are not mentioned.

Of course, the more general the applicability, the more interesting the result, as the psychologists very well know : they say "X correlates with Y and Z in adult humans", and not "X correlates with Y and Z in this set of 58 adults" that they used to infer that particular correlation. The latter finding will not be published in high-profile journals.

Fully specifying the context is certainly more difficult in the social sciences than in physics, but the solution is adding more rigour, not claiming for psychologists the right to generalize their results arbitrarily (what holds for a few dozen psychology undergraduates in an American research university may not apply to all mankind). If the generalisation is shown to be incorrect, this is not proof of a new and exciting result, but simply a sign that the original authors oversold their findings.