HARKing in a bachelor thesis - I feel tormented

[Carl-Richard]

So I had two hypotheses in my bachelor thesis: 

H1: mindfulness is correlated with physical activity

H2: controlling for intrinsic motivation towards exercise weakens the correlation between mindfulness and physical activity, which could indicate that the effect of mindfulness on physical activity is mediated by intrinsic motivation towards exercise.

I ran the analyses, and none of the hypotheses had a statistically significant result (which means both hypotheses get scrapped). Then right after running the analyses, I had the realization that running an analysis for another hypothesis (H3: "mindfulness is correlated with intrinsic motivation towards exercise") is the logical next step after scrapping H1 and H2, because it could add support to a moderation relationship as opposed to a meditation relationship (mindfulness moderates the relationship between intrinsic motivation towards exercise and physical activity).

The stupid thing though is that I had this realization right after running the analyses for the other hypotheses, which means that H3 becomes a "post-hoc" hypothesis, and presenting it alongside the other hypotheses would be considered HARKing ("“presenting a post hoc hypothesis in the introduction of a research report as if it were an a priori hypothesis”), which is considered an ethically dubious practice.

There are many types of HARKing, and there are many different views on what types of HARKing are considered OK, and there is this one view that says that this particular scenario is an OK form of HARKing (because it's justified based on the theoretical background of the thesis), and also, my advisor says that I could do it and it would be OK by him, but I still feel tormented by it. If I had only spent one second thinking through the consequences of scrapping H1 and H2 before running the analyses, I would virtually certainly have spotted this extremely obvious H3, and I would not be in this situation.

So yes, I feel tormented by it even though my advisor has green-lighted it, so am I being too neurotic? Do you think a very strict examiner would penalize me for going along with it? (I'm of course going to disclose the whole decision-making process behind H3, which is called "THARKING" — a transparent form for HARKing, which absolutely minimizes the ethical dubiousness). I don't want to poison my work with ethical misconduct ? I would truly appreciate some advice  

Edited March 15, 2023 by Carl-Richard

[Carl-Richard]

I think I've solved it for now. My thesis is going to be a weird Frankenstein monster of combining the principles of Grounded Theory and quantitative analysis.

[Nilsi]

Instead of this nonsense, you should spend your time thinking about how to create methodology that isnt so liable to manipulation and perverse incentive. 

Academic science is such a shitshow, its incredible.

[Carl-Richard]

49 minutes ago, Nilsi said:

Academic science is such a shitshow, its incredible.

I don't see much alternatives though

[Nilsi]

Just now, Carl-Richard said:

I don't see much alternatives though

Become a rogue scientist and study whatever you want to study without being tied to institutional corruption and archaic methodology.

[Carl-Richard]

32 minutes ago, Nilsi said:

Become a rogue scientist and study whatever you want to study without being tied to institutional corruption and archaic methodology.

Who will pay you? OnlyFans subscribers?  But I agree, I've always seen academia as the second best thing.

Edited March 17, 2023 by Carl-Richard

[Nilsi]

8 minutes ago, Carl-Richard said:

Who will pay you? OnlyFans subscribers?  But I agree, I've always seen academia as the second best thing.

Figure it out! - I thought you were an idealist?!

We live in the 21st century. There has to be some way to monetize this. 

Creating a personal brand and building an audience on social media would be a low hanging fruit, but I'm sure there are dozens of ways to pull this off.

Edited March 17, 2023 by Nilsi

[Devin]

@Carl-Richard      But h3 wasn't made by seeing the trend in the other analyses, you had to do a 3rd analyses to prove it, you sure it's considered HARKing? The way I read it HARKing would be running your first two analyses, seeing a pattern that you write a hypothesis on of which you can use that actual already existing analysis to prove.

https://en.m.wikipedia.org/wiki/Post_hoc_analysis

Edited March 17, 2023 by Devin

[Devin]

HARKing (hypothesizing after the results are known)

You didn't know the results did you?

[Nilsi]

10 minutes ago, Devin said:

@Carl-Richard      But h3 wasn't made by seeing the trend in the other analyses, you had to do a 3rd analyses to prove it, you sure it's considered HARKing? The way I read it HARKing would be running your first two analyses, seeing a pattern that you write a hypothesis on of which you can use that actual already existing analysis to prove.

https://en.m.wikipedia.org/wiki/Post_hoc_analysis

This is like drawing the target around the arrow.

[Devin]

Just now, Nilsi said:

This is like drawing the target around the arrow.

Doesn't read like he did that to me.

To me he just took three shots at the target.

[Nilsi]

1 minute ago, Devin said:

Doesn't read like he did that to me.

To me he just took three shots at the target.

The problem being that he only had two shots.

[Devin]

4 minutes ago, Nilsi said:

The problem being that he only had two shots.

I don't think so.

[Carl-Richard]

6 hours ago, Devin said:

@Carl-Richard      But h3 wasn't made by seeing the trend in the other analyses, you had to do a 3rd analyses to prove it, you sure it's considered HARKing? The way I read it HARKing would be running your first two analyses, seeing a pattern that you write a hypothesis on of which you can use that actual already existing analysis to prove.

https://en.m.wikipedia.org/wiki/Post_hoc_analysis

H3 was considered relevant when I realized the consequences of scrapping H1 and H2. Let me try to explain with my sleepy brain:

Ruffault et al. (2016) conducted a study where they found that mindfulness moderates the relationship between two variables: 1. intrinsic motivation towards exercise (independent variable), and 2. physical activity levels (dependent variable). So mindfulness increases the effect that intrinsic motivation towards exercise has on physical activity levels, but since it's a moderation effect, mindfulness does not have to be correlated with physical activity levels for this effect to exist.

That was in fact the reasoning that Ruffault et al. (2016) used for choosing the moderation model in the first place: they couldn't find a correlation between mindfulness and physical activity, but they did find a correlation between mindfulness and intrinsic motivation towards exercise, and they of course found a correlation between intrinsic motivation towards exercise and physical activity.

On the other hand, had they found a correlation between mindfulness and physical activity, they would've most likely ran a mediation model instead, because mediation is when you have an effect on the dependent variable, but it's mediated by an intermediate variable (i.e. mindfulness' effect on physical activity would be mediated by intrinsic regulation towards exercise). These are subtle differences, but this lead me to my insight about the consequences of scrapping H1 and H2:

When you scrap H1 and H2, you have excluded the possibility of a mediation relationship, mainly because mindfulness is not correlated with physical activity. But, it's still possible to find support for a moderation relationship. Why? Because the criteria for a moderation relationship in this case is that mindfulness must be correlated with intrinsic motivation towards exercise (H3), and intrinsic motivation towards exercise must be correlated with physical activity (which was already established in the regression model used for answering H2).

The only reason I consider it HARKing has to do with the timing: I ran the analysis, which when I saw the results, made me have to scrap H1 and H2, and then I had the insight. But I could've easily predicted the consequences of scrapping H1 and H2 before running the analysis. It was just me being bad at forward-thinking, and I needed to actually be in the situation of scrapping H1 and H2 in order to prompt me to think about the consequences of that scenario. Anyway, it doesn't matter now as my plan is to be transparent about the whole process and do it under the guise of "Grounded Theory" (which was one of the suggestions from my advisor). That option is so much more freeing than having to shoehorn myself into a more standard format.

This is probably way too dense, but fuck it, I'm exhausted from writing my last bunch of clingy e-mails to my advisor lol. Also, post-hoc-ing the H3 wasn't the only problem. I ran a second survey on a different population after seeing some of the results of the first population just out of curiosity (because I saw some patterns in the data that made me ask a very interesting question), and the data from that survey came in much more handy than I expected (again, lack of forward-thinking), and I would feel bad if I had to exclude that data (even though I could easily exclude them). They call me the Zentific Devil.

The interesting question I asked myself was: "it seems like many people scored 0 on physical activity in the spirituality-related population, so what if mindfulness doesn't make you "take up" exercise, but instead it just increases exercise frequency in people who already exercise?", hence I made a survey for people who exercise frequently in order to avoid 0-scores and posted it all across lifting-related subreddits and got the same amount of participants in just a few days. And yes, that population indeed turned out to have zero people with 0-scores on physical activity). I just recently did a data cleaning protocol which significantly improved the amount of ugly outliers, so I would have to re-run the comparisons between the two populations to see if there is a difference.

Edited March 18, 2023 by Carl-Richard

[Devin]

@Carl-Richard Thanks for the background I was actually curious and wanted to ask about it. I still think you're being hard on yourself for what seems to me to be no reason, but I'm glad you've settled on a course of action.

    Do you ever use "proportional", "integral", "derivative", with this stuff? "Mediation" and "moderation" are too close in English, I had to keep referring back, I'm sure the jargon gets engrained easily, but man those words are very similar.

     I have a friend that is a statistician, has a masters in statistics if I happen to see her I would definitely be getting her opinion for you as well, she was a teaching assistant too, maybe there's something I don't understand but it seems fine to me.

Edited March 18, 2023 by Devin

[Carl-Richard]

8 minutes ago, Devin said:

@Carl-Richard Thanks for the background I was actually curious and wanted to ask about it. I still think you're being hard on yourself for what seems to me to be no reason, but I'm glad you've settled on a course of action.

    Do you ever use "proportional", "integral", "derivative", with this stuff? "Mediation" and "moderation" are too close in English, I had to keep referring back, I'm sure the jargon gets engrained easily, but man those words are very similar.

     I have a friend that is a statistician, has a masters in statistics if I happen to see her I would definitely be getting her opinion for you as well, she was a teaching assistant as too, maybe there's something I don't understand but it seems fine to me.

No, I haven't seen those terms used in any of our statistics books or in any of the studies I've been looking into in my thesis.

 

We don't really use "mediation" and "moderation" in my language (Norwegian). If I were to translate it directly, it would be "indirect effects" vs. "interaction effects".

Edited March 18, 2023 by Carl-Richard

[Devin]

14 minutes ago, Carl-Richard said:

No, I haven't seen those terms used in any of our statistics books or in any of the studies I've been looking into in my thesis. We don't really use "mediation" and "moderation" in my language (Norwegian). If I were to translate it directly, it would be indirect effects vs. interaction effects.

       I'm not really familiar with statistics, mediation and moderation are the terms they use here I had googled them, but the Norwegian translation would be easier for a layman like me to keep them straight. My terms were more calculus.

Edited March 18, 2023 by Devin

[Carl-Richard]

21 minutes ago, Devin said:

       I'm not really familiar with statistics, those are the terms they use here I had googled them, but the Norwegian translation would be easier for a layman like me to keep them straight. My terms were more calculus.

I basically didn't know anything about this before like a month ago ?. I had to read the final chapters of the stats book that weren't included in the curriculum and watch a bunch of videos on how to do hierarchical linear regression in JASP before I felt confident enough to start writing my thesis.

Something that I really struggled with when reading about regression with multiple variables was not getting lost in trying to understand how the hell the computer "controls for" multiple variables in a regression model. That is why we let the computer do it, because your brain can't really comprehend it.

Edited March 18, 2023 by Carl-Richard

[Carl-Richard]

On 18.3.2023 at 4:05 AM, Sultan Al Hind said:

@Carl-Richard @Carl-Richard u r so smart. what is the secret . is it inborn or did u develop it

Firstly, that's flattering ? Secondly, I don't like the word "smart". I think passion is a better word.

If there is something I'm passionate about, I'll do it a lot and try to do it the best I can. The "smartest" thing you can do is to find what you're passionate about and do that. If you don't know what you're passionate about, just try something out, challenge yourself. Who you are now is not set in stone forever. You can adapt and learn new things and it becomes a part of you. 3-4 years ago, I didn't know I would be doing what I'm doing now, but I enjoy it.

Also, there is always compromise. I've considered three paths that most deeply reflect myself throughout my life: musician, scientist, and spiritual teacher. I'm constantly sort of fluctuating between all of them, but there have been different phases where I've been focusing more on one over the others, and I feel that will continue into the future.

Edited March 21, 2023 by Carl-Richard

[Sultan Al Hind]

@Carl-Richard ooo man . thanks for sharing.  keep it up. Keep up the good work