Contextualizing Methodological Problems Within the Social Sciences

[DocWatts]

If anyone here actually works within (or is broadly knowledgeable about) one of the Social Sciences, I'd be highly interested in hearing your insights on the matter.

I'll often see critiques on works meant for the Public (ie non specialists in that academic field) refuted not by a broad refutation of the work's overall argument, but death by a thousand cuts style critiques which cite methodological problems within the research used to back up arguments in the work. 

Steven Pinker's work comes to mind for a good example of this, in regards to Better Angles of Our Nature. But I've also seen this applied to a lesser extend for some of Jared Diamond's work as well. I could also list the numerous examples from within the field of psychology, with the replicability problem that's plagued the field and has supposedly discredited a number of high profile studies.

As to contextualizing these types of critiques, do methodological problems on their own present a good case for questioning someone's entire argument, or should these also be backed up my other forms of critique, such as argumentation? Obviously this will vary on a case by case basis to some degree, and will also depend on how heavily the author's argument depends on empirical research, but in general how much weight should be afforded to these types of arguments? Or are they more often Red Herrings if not backed up by other forms of critique?

Not a Social Scientist or Researcher myself, so it would be interested to hear some other thoughts on this issue.

Edited February 7, 2021 by DocWatts

[Leo Gura]

Way too broad. You have look at the details of every case. Science is very technical by its nature.

[Carl-Richard]

I think I have something in this ballpark, more specifically about construct validity and internal critique within a field (as opposed to critique from the outside).

There is this book on personality psychology that goes into why trait models like Big Five are preferred over typology models like MBTI. It basically boils down to the fact that statistically speaking, people generally don't fall into different "types" of personalities (introverted, extroverted etc.) but rather that these qualities follow a normal distribution in the population. If the traits instead followed a bimodal distribution, then that would be more in line with a typology model.

In other words, very few people are extremely introverted or extroverted (most people are somewhere in the middle). The trait model says that you can be either high or low in a trait instead of being simply either/or, which seems to be a more accurate representation of the data. This is a specific argument that the book mentioned about why typologies are generally not well-received in psychology.

Now, there is no reason to dispute the actual facts of the matter, but here are some few observations:

1. Most people when operating in the world, don't use statements like "I'm a person high in introversion". They will most likely say "I'm an introvert". Therefore, it doesn't really matter which type of model you feed people, because you'll most likely end up with a typology description either way.

2. Some MBTI personality tests do infact give you a statistical distribution of a trait, and the type you end up is then simply reverse engineered from those numbers (if you're 60% introvert, you'll have that as your type).

Therefore, at the end of the day, it's only a difference in description and representation. If you were to describe something typologically while using numbers that don't follow a bimodal distribution, that doesn't actually negate the numbers. It's only a different description, less accurate yes, but a valid description nonetheless. 

In my own opinion, as long as you're aware of these things, you should be able to say "I'm high in introversion" AND "I'm an introvert" based on any model (typology or trait). Given the same data, it shouldn't really be a cause for concern.

Even if typologies were somehow inherently less equipped with dealing with reality than trait models, I still embrace the "let a hundred flowers bloom" approach, because I ultimately see science through an utilitarian lens and not the absolutist "science=truth" lens

Edited February 8, 2021 by Carl-Richard

[DocWatts]

@Carl-Richard That's interesting, the Construct Validity framework that you bring up seems to make a lot of sense for some of the critiques I see of pop-psychology models (such as Myers-Briggs). Sometimes it can be hard to parse out well grounded criticism pointing out the flaws of particular model from Snobbery on the part of Academics, as any model presented to non specialists (ie the Public) is going to have to be 'flattened' to some degree, losing some of its nuance in the process. 

I'm also of the Utilitarian mindset when it comes to Science, while at the same time I also see that ideas can be over-simplified to the point of becoming misleading, thus creating a potential to cause harm by propagating misinformation.

[Carl-Richard]

@DocWatts The general argument against typologies concerning construct validity is really just a drop in the ocean of why MBTI doesn't measure up to Big Five (empirical evidence, reliability, trait independence, comprehensiveness, etc.). I think the reason why the book spent so much time explaining it was probably just to teach about bimodal and normal distribution. I didn't consider that perspective first time I read it so I may have overreacted a little bit