A suspicion I had about the field during my bachelor that turned out to be a big deal

[Carl-Richard]

So when I was writing my bachelor thesis in psychology on the relationship between mindfulness and physical activity, I came across two cases where a single study tried to measure the same thing (mindfulness) using two different questionnaires and came to very different conclusions for each questionnaire (i.e. mindfulness correlates positively with physical activity vs. mindfulness does not correlate with physical activity).

This made me have a moment of "what even is science?". Well, it turns out that my feeling in that moment seems to be echoed in a bunch of articles I've had to read in these past weeks, largely specific to the social and behavioral sciences (and therefore psychology). The articles are mostly addressing possible causes of the replication crisis (for some numbers: depending on the sub-field, some estimates say 50-75% of studies in psychology fail to replicate).

Here are some excerpts from one article to show you what I'm talking about:

Quote

[...] In practice, however, the majority of psychologists have no compunction about verbally generalizing their results not only to previously unseen subjects, but also to all kinds of other factors that have not explicitly been modeled— to new stimuli, experimenters, research sites and so on. Under such circumstances, it’s unclear why anyone should really care about the inferential statistics psychologists report in most papers, seeing as those statistics bear only the most tenuous of connections to authors’ sweeping verbal conclusions. [...]

[...] One perfectly reasonable course of action when faced with the difficulty of extracting meaningful, widely generalizable conclusions from effects that are inherently complex and highly variable is to opt out of the enterprise entirely. [...]

[...] It should not be terribly surprising if few tenured professors are eager to admit (even to themselves) that they have, as Paul Meehl rather colorfully put it, “achieved some notoriety, tenure, economic security and the like by engaging, to speak bluntly, in a bunch of nothing” (P. E. Meehl, 1990b, p. 230). [...]

"The generalizability crisis" - https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/generalizability-crisis/AD386115BA539A759ACB3093760F4824

 

Reading those last two sentences took me straight back to looking at those studies a few months ago. It made me feel that the entire quantitative branch of psychology was sort of meaningless. "Just quit!". Will I? What if psychology can be saved? Is it all for nothing? What do you think?

 

Here is another article on the topic:

"Addressing the theory crisis in psychology" - https://link.springer.com/article/10.3758/s13423-019-01645-2

Edited September 11, 2023 by Carl-Richard

[Sucuk Ekmek]

Hey, I dont know anything about psychology, but this sounds like you are actually doing science!

[Carl-Richard]

4 hours ago, Sucuk Ekmek said:

Hey, I dont know anything about psychology, but this sounds like you are actually doing science!

That is the problem, isn't it? Does it only sound like we're doing science or are we actually doing science?

[Osaid]

People just suck at observing things.

Is it a science to ask people what they think? I mean it can be, but you gotta take it with a grain of salt.

I'd argue social sciences can be a thing, they are a thing, but trying to make conclusions from it is JUST based on what people think. It is limited to that. It's not a failure just a bad interpretation of what you gathered.

I don't think it's coincidence that it has become a big thing though, because it seems like something is there that people are trying to point to. So maybe quantitative science can be like a finger pointing at something and saying "hey this needs to be investigated."

Hope what I said made any sense.

[LSD-Rumi]

@Carl-Richard I don't care about studies in psychology that much. It is Essential in psychiatry but not that important in psychology. Proper psychological understanding depends on explainations not studies, so it is more of a philosophical pursuit rather than based on research. 

Great psychologists are the ones who understand how humans think and the inner workings ofnthe subconscious mind.

[Carl-Richard]

10 hours ago, Osaid said:

People just suck at observing things.

Is it a science to ask people what they think? I mean it can be, but you gotta take it with a grain of salt.

It's funny you say that, because some critique from another paper I've read talked about how most psychology research, partially due to the relative death of behaviorism and birth of cognitive psychology, has shifted away from actually observing behavior and over to trying to explain behavior using some theoretical mechanism and measuring that mechanism using some indirect measurement like a questionnaire. So the question then becomes: is psychology any longer a study of behavior, or is it a study of speculations about mechanisms behind behavior and indirect measurements of said behavior?

 

10 hours ago, Osaid said:

I'd argue social sciences can be a thing, they are a thing, but trying to make conclusions from it is JUST based on what people think. It is limited to that. It's not a failure just a bad interpretation of what you gathered.

10 hours ago, LSD-Rumi said:

Proper psychological understanding depends on explainations not studies, so it is more of a philosophical pursuit rather than based on research. 

Mmm. The paper that talks about the "generalization crisis" says that one possible course of action for psychology is to move away from quantitative approaches and embrace qualitative approaches, i.e. approaches that don't rely on inferential statistics for their conclusions. So the field limits itself to descriptive statistics (e.g. "the study looked at 20 women in a school setting"), makes interesting observations and verbal conclusions (e.g. "these people think x, these other people think y"), and avoids conclusions like "the p-value is less than .05, therefore there is a statistically significant correlation between the two variables, and therefore our hypothesis has been corroborated".

The problematic part about the latter that causes the generalization crisis is the inferential jump from "there is a significant correlation" to "our hypothesis has been corroborated", because the specific conditions that produced the statistics (the research design; the stimuli, experimenters, measurement instruments, etc.) are usually not mentioned in the hypothesis. Hence there is a tendency to generalize from one research finding to a larger conclusion when that is not actually warranted.

But you might say that if we choose to only make highly specific conclusions, e.g. "there was a significant correlation in this setting, given these research parameters, etc.", we can avoid the generalization problem, but this puts the cart before the horse in the sense that we still use these generalized hypotheses to drive our research and focus in on what we think is relevant. If we do continue making research questions around a general hypothesis but only report highly specific conclusions that never actually address the general hypothesis in any satisfying way, then we're never getting what we're after. We're instead fooling ourselves in a way, doing what Feynman called "cargo cult science"; thinking we're doing important work because the cool-looking statistics seem to add up, but it never actually points to any important conclusion.

So what both of you said seems to point in that direction of embracing qualitative over quantitative research, i.e. merely inquiring into some behavior or asking somebody for their experiences without relying on any hard statistics for your conclusions, a bit like philosophy. That said, there are attempts at improving the quantitative approach which may help save it somewhat, but I'll maybe save going into that for later.

Edited September 12, 2023 by Carl-Richard

[Osaid]

32 minutes ago, Carl-Richard said:

is psychology any longer a study of behavior, or is it a study of speculations about mechanisms behind behavior and indirect measurements of said behavior?

It's a measure of what people think their behavior is, not behavior. It's indirect. And it's absolutely not behavior itself. So it's really important to keep that nuance. So yeah, the latter.

32 minutes ago, Carl-Richard said:

So what you of both said seems to point in that direction of embracing qualitative over quantitative research, i.e. merely inquiring into some behavior or asking somebody for their experiences without relying on any hard statistics for your conclusions, a bit like philosophy.

It becomes a big red herring I think, and I think this is the right direction.

Like, recently we had a post here about college leading to more depression or whatever, but like, depression also occurs outside of college. So the college setting becomes a red herring. It's not a college issue. It's something more fundamental that's not being addressed. 

Changing the college system could mediate it or reduce depression, but it doesn't actually touch on depression at all, it just touches on the college system, and then depression is reduced as a side effect. Which is good, but it has not much to do with depression. Many people would look at this and say "college causes depression" or "this aspect of college is causing depression." This turns into a rabbit hole of chasing causes which are inferred from previous causes and so on.

Imagine trying to measure the behavior of a black hole by measuring the gravity of the Earth. And then inferring the gravity of the black hole purely from the gravity of Earth. This is the same indirectness and leeway being given to quantitative behavioral science, but this discrepancy is shrugged off by most because there's this bias when it comes to being quantitative that is like "If we relate it to an actual object, it becomes more true and objective."

And, in you or some scientist reading this, there might be this fear that comes up which is saying "But then, we can't measure anything! Science is lost! This is stupid because now we can't do science anymore!", and this very clinging to science is purely just dogma. I am not saying that this fear or sentiment is true, but rather, you should just be completely open to this possibility, and it shouldn't be a fear. As a scientist, you should be completely open to the idea that science might not work here, or that certain things just can't be measured. Or, as an even better scientist, you should be inspired by this to try and revolutionize science by seeing its limits. I'm reminded of this Niels Bohr quote: "How wonderful that we have met with a paradox. Now we have some hope of making progress"

Edited September 11, 2023 by Osaid

[Yimpa]

37 minutes ago, Carl-Richard said:

So what you of both said seems to point in that direction of embracing qualitative over quantitative research, i.e. merely inquiring into some behavior or asking somebody for their experiences without relying on any hard statistics for your conclusions, a bit like philosophy. That said, there are attempts at improving the quantitative approach which may help save it somewhat, but I'll maybe save going into that for later.

This is where AI will have a major role in the future. Might as well leave all the logical stuff for the AI to handle. 
 

 

[Carl-Richard]

8 hours ago, Yimpa said:

This is where AI will have a major role in the future. Might as well leave all the logical stuff for the AI to handle. 

My dude, AI can't even tie its own shoelaces. And when it can, it will probably throw psychology in the garbage.

 

8 hours ago, Osaid said:

Imagine trying to measure the behavior of a black hole by measuring the gravity of the Earth. And then inferring the gravity of the black hole purely from the gravity of Earth. This is the same indirectness and leeway being given to quantitative behavioral science, but this discrepancy is shrugged off by most because there's this bias when it comes to being quantitative that is like "If we relate it to an actual object, it becomes more true and objective."

That's a brilliant way of putting it. It does feel exactly like that sometimes.

 

8 hours ago, Osaid said:

And, in you or some scientist reading this, there might be this fear that comes up which is saying "But then, we can't measure anything! Science is lost! This is stupid because now we can't do science anymore!", and this very clinging to science is purely just dogma. I am not saying that this fear or sentiment is true, but rather, you should just be completely open to this possibility, and it shouldn't be a fear. As a scientist, you should be completely open to the idea that science might not work here, or that certain things just can't be measured. Or, as an even better scientist, you should be inspired by this to try and revolutionize science by seeing its limits. I'm reminded of this Niels Bohr quote: "How wonderful that we have met with a paradox. Now we have some hope of making progress"

So yeah, on that note of revolutionizing science, like I was alluding to earlier, when it comes to finding better solutions within the quantitative approach, you have things like Registered Replication Reports (RRRs), which is when a large group of scientists make a coordinated effort to create direct replications for a selection of studies that seem important to the field.

The author in the "generalization crisis" paper problematized this by saying that direct replications don't actually matter with respect to the conclusions if the original findings over-generalized their findings, so there needs to be a similar effort for so-called "conceptual replications" (i.e. where instead of keeping the research design fixed like in direct replications, you intentionally vary the research designs to see where the correct level of generalization lies).

You also have efforts like the Psychological Science Accelerator (PSA) that work similarly to RRRs by more generally coordinating research around a select set of topics. One of the critiques against psychology as a field is the lack of "cumulative research", i.e. research that gets worked on over time and gathers a solid empirical basis. The alternative is often that some research gets started and then forgotten when people find out it's bunk, which leads to a constant cycle of jumping from hypotheses to hypotheses in a way that severely fragments the field.

It's uncertain to which degree it's possible to lessen the fragmentation of psychological research, because the human mind is in a very fundamental sense multi-faceted. This then feeds into the "paradigm debate" (about whether psychology should be considered a single paradigm or multiple paradigms); something which Kuhn himself used to conclude that psychology was not a real science, because he thought a real science consists of a shared paradigm that creates a focus around cumulative research).

As for securing the quality of research in more applied settings (e.g. COVID-19), you have proposals like «Evidence Readiness Levels» (ERLs) inspired by NASA's "Technology Readiness Levels" (TRLs) for testing technology related to aeronautics and space.

Edited September 12, 2023 by Carl-Richard

[Yimpa]

2 minutes ago, Carl-Richard said:

My dude, AI can't even tie it's own shoelaces. And when it can, it will probably throw psychology in the garbage.

Just as Mother Nature intended 

[Sucuk Ekmek]

4 hours ago, Carl-Richard said:

That is the problem, isn't it? Does it only sound like we're doing science or are we actually doing science?

Yeah concerning our  and its nature  doing science is challenging, but there should be one actually doing it.

[StarStruck]

Psychology is NOT a real science to begin with. It is a soft science and it can't be compared to the hard sciences. It is subjectivism through the lens of objectivism. Wilber has some great things to say about this.

[Carl-Richard]

49 minutes ago, StarStruck said:

Wilber has some great things to say about this.

Which is?

[StarStruck]

19 minutes ago, Carl-Richard said:

Which is?

It is written in his book Integral psychology but it was a year ago I read his book. 

Edited September 11, 2023 by StarStruck

[DocWatts]

I sometimes wonder if the replication crisis is a consequence of the inherent limitations of trying to apply empiricism to a qualitative domain. People aren't deterministic entities, so it's hardly surprising that an outside-in approach to why we do the things we do is going to be messy (mind you that doesn't necessarily mean invalid; just messy).

Edited September 11, 2023 by DocWatts

[zurew]

22 hours ago, Carl-Richard said:

Reading those last two sentences took me straight back to looking at those studies a few months ago. It made me feel that the entire quantitative branch of psychology was sort of meaningless. "Just quit!". Will I? What if psychology can be saved? Is it all for nothing?

Thats sounds like someone who is contemplating about quitting doing psychology.

Are you contemplating about changing majors?

 

Btw, I know this is worthless to mention(because its almost obvious), but there has to be some good philosophy of science material that at least attempts to address the replication crisis and maybe more specifically the issue, that you've brought up.

 

5 minutes ago, DocWatts said:

People aren't deterministic entities, so it's hardly surprising that an outside-in approach to why we do the things we do is going to be messy (mind that doesn't necessarily mean invalid; just message).

Thats a very interesting insight into this problem.

[Carl-Richard]

5 minutes ago, zurew said:

Thats sounds like someone who is contemplating about quitting doing psychology.

Are you contemplating about changing majors?

I've always contemplated quitting, but not so soon. Regardless, while my degree is technically a psychology degree, it also specializes in neuroscience, and there are many potential avenues there. I think my peers who're specializing in social and cognitive psychology are more worried than us neuroscience guys 

 

10 minutes ago, zurew said:

Btw, I know this is worthless to mention(because its almost obvious), but there has to be some good philosophy of science material that at least attempts to address the replication crisis and maybe more specifically the issue, that you've brought up.

There are. It's just that some of the critiques seem to hit on something deep that might be incredibly hard to solve. At the same time, some parts of the field are less affected by things like the replication crisis than others (for example, the literature on the benefits of brain training techniques, which I'll go do now actually ; I'm currently doing Triple 3-Back ).

[Carl-Richard]

49 minutes ago, StarStruck said:

It is written in his book Integral psychology but it was a year ago I read his book. 

You don't remember anything though?  

[zurew]

18 minutes ago, Carl-Richard said:

than us neuroscience guys 

In that case, I will pick your brain for some insights about neuroscience in the future .

45 minutes ago, Carl-Richard said:

(for example, the literature on the benefits of brain training techniques, which I'll go do now actually ; I'm currently doing Triple 3-Back ).

The dual n-back stuff is interesting. I have seen  some claims about it having the potential to improve fluid intelligence. Is that actually true based on your research on the subject?

[Carl-Richard]

2 hours ago, zurew said:

The dual n-back stuff is interesting. I have seen  some claims about it having the potential to improve fluid intelligence. Is that actually true based on your research on the subject?

I'm literally only basing what I said on a PowerPoint slide that my professor used when talking about the extent of the replication crisis, and it mentioned among other things that "brain training techniques" is sort of in the clear (not N-Back training specifically), so don't read too much into that. I still feel that it works (very scientific hehe). I thought about looking into it more (and I probably should given my concerns ).

 

4 hours ago, StarStruck said:

Psychology is NOT a real science to begin with. It is a soft science and it can't be compared to the hard sciences. It is subjectivism through the lens of objectivism. Wilber has some great things to say about this.

For the record though, it seems like all sciences that touch on human behavior in some form or another is implicated in the replication crisis, not just psychology: sociology, economics, biology, medicine, etc. But again, it's not all equally implicated, even within psychology. For example, some estimates say that cognitive psychology has a replication rate of 50% while social psychology only has 25%. And then you should also expect some findings to be more robust than others, especially the fundamentals of any given field (e.g. the fundamental workings of memory and perception in cognitive psychology; working memory, attention, etc.). It's usually when the hypotheses get more specific and novel that you run into problems like replicability and generalizability.

Edited September 12, 2023 by Carl-Richard