I completely agree, and I'm glad someone's on this thread reminding everyone about the terrible state that social science research is in.
Human beings are complicated and difficult to study, so I can appreciate the challenge that social science researchers have in front of them. But, propagating conclusions from a field of research that has a 33% reproducibility rate is just negligent journalism. It's not true just because it's in one study and it matches our intuition or something we read in a Malcolm Gladwell book.
> Human beings are complicated and difficult to study
Well that's a positive way to put it. I wonder whether it is possible to find steady cause and effect patterns at all.
Same goes for economy (which IS basically a proxy for human behavior). You can certainly have a go at documenting its history but the present and future are probably not predictable because they are subject to evolutionary responses.
When I had to take an economics 101 class negative interest rates were out of the question and look what we have now.
Same here, especially when the study surveys ordinary workers. People who run larger companies and banks typically score higher on the psychopathy scale (see the documentary "I am Fishhead"). Just because individuals may desire autonomy doesn't mean those type of people make it into leadership roles.
Wait, what? You base your argument for disagreeing with the study on a documentary? You're aware that as bad as the standards for social sciences are, they still run circles around the standards for documentaries, right?
And you're aware that the core part of that documentary is based on an (AFAIK) unreproduced study on psychopathy - by somebody who likely has a vested interest in psychopathy being a problem. (Robert Hare - see www.hare.org)
I'm all for questioning the accuracy of studies, but I'd hope we'd use somewhat higher standards than that.
The original paper was proven to be more alarmist than true. Dan Gilbert's team put out an excellent analysis of the paper and found many poorly performed replications. That being said, it sheds light on the greater problem of p-value hacking that may exaggerate any kind of statistical result
If it is the article I remember, one example was showing that the study could be replicated if done at the same university (which would be years after the first study), but the criticism was that it couldn't be replicated at a different university.
So the criticism of the criticism seems to be quite overstated. If the study can only be replicated at a single university (or even all in a general area), it means that we need to be very careful in generalizing the results. So to the extent that people take a study of one very specific group and generalize to an entire culture (or worse, to all humans), the criticism sticks.
Back when I was getting my psychology degree, there was a joke that 90% of the findings in psychology can only be generalized to broke American college students who are willing to be lab rats for beer money. While that overstates the problem, there is still a lot of truth in it.
Obviously it's not possible to pick a sample size large enough to eliminate all bias. However it's important to pick a sample that doesn't inherently bias the effect that is being measured. This has to apply to both the original and the replication.
The response to the paper points out few examples where the replication was not judicious in picking a sample that minimized its influence on the measured effect. Even besides that, the original study had sampling and scoring procedures which contradicted its original aims.
Lastly, sensationalist headlines of "more than half of all papers are not replicable" are blatantly wrong and not what the original paper even states.
As to your point, I agree that only being able to replicate in one sample type is grounds for skepticism, but more replications have to be done to ensure the replication sample is not the one that is flawed. One can't really draw conclusions unless the replication is repeated a large number of times across a wider sample space.
>However it's important to pick a sample that doesn't inherently bias the effect that is being measured. This has to apply to both the original and the replication.
All this shows is that psychology is less useful that people thing because any data is likely only applicable to the given group. Even if we say the studies are still good, this means that psychology loses the ability to be generalized to others groups beyond the one studied. And given how often psychology has that done to it, we must question every finding all the more.
I get really sceptical about these social sciences studies, especially after the recent round of replication attempts that failed.