The replication crisis in psychology has been a fascinating lens into the methodology of many well known professionals. See this bit from Conversations with Tyler where Cowen is talking with noted psychologist Daniel Kahneman:
COWEN: Now your basic distinction between System 1 and System 2, thinking fast and thinking slow — to the extent that particular results do not replicate, do you view that as undercutting the System 1 versus System 2 distinction? Or is that immune to the degree of replicability?
KAHNEMAN: There were whole sets of results that I published in Thinking, Fast and Slow that I wish I hadn’t published because they’re not reliable.
Whether it undercuts . . . The idea of two systems is really anchored in a basic sort of fact of experience, that the process by which you get 2 plus 2 is fundamentally different from the way that you get 17 by 24. One of them happens automatically, associatively, quickly. You have no control. The other demands effort and is slow and so on. That’s immune to replication.
COWEN: But if there’s a bias in individuals and noise, why should we trust our experience about this apparent sense of having two methods? Is it three? Is it four?
KAHNEMAN: Well, in the first place, those are extremes. It doesn’t mean that there aren’t others. It doesn’t mean that there is not a continuum. But there is at least a continuum to be explored of those two extremes. Of that I’m quite confident.