Research Methods in Cognitive Science

# Research Methods in Cognitive Science
## Week 4: The Credibility Revolution or Why We Are Not F***KED
### Jason Geller, Ph.D.
### 2021-09-22

---

- To see if you read, if you get the question right you will get 2 extra credit points

---
# Outline

- The crisis in science

- Last Week Tonight with John Oliver
  - Warning: Language and Foul Humor

- What is replication?

- The history of the replication crisis in psychology

- The issues that lead to the crisis

- Simmons et al. (2011)

- Potential Solutions

- Open Science 
  - Open Science Framework
  - Preregistration
  - ORCID
---
exclude: true
# Poll

https://www.youtube.com/watch?v=0Rnq1NpHdmw&t=666s

---
# Replication

<img src="wizard.png" width="70%" style="display: block; margin: auto;" />
--

- Replication can be successful

- Replication can be unsuccessful 
---

---
# The Replication Crisis: A Timeline

- A little history

- 2011: Daryl Bem published a paper in JPSP (top journal) claiming to have found evidence that ESP exists

- PsychoPy Demo 
  
  - 2011: Diederik Stapel busted for fabricating data (**58 papers retracted to date**)

- 2011-present: Major findings (psychology) not replicated
  
     - E.g., Ego depletion, embodied cognition, power posing, social priming

- 2015: Open Science Collaboration tried to replicate 100 psychology studies
---
# Open Science Collaboration (2015)

.pull-left[- Open Science Collaboration attempted to replicate 100 studies published in 3 top psychology journals in 2008

- Replications used materials supplied by original authors and were high-powered

- Results: 39% of the original studies were successfully replicated

- 25% of social psychology studies replicated
      - 50% of cognitive psychology studies replicated
          - Effect sizes overestimated in original studies
]

]

---
# Why?

# Do we have a problem?

---
# False Positive Psychology (Simmons, Nelson, & Simonsohn, 2011)

- False positives are Type 1 errors

- Claim an effect exists when it actually does not
  
- Simmons, Nelson, & Simonsohn (2011) demonstrated that it’s easy to statistically support a hypothesis that is actually false (i.e., find a false positive)

---
# The Dangers of False Positives

- Can be dangerous
  - E.g., health research: claim that a treatment is effective when actually it isn’t, or even has negative side effects
  
- Wastes resources

- Researchers waste time, effort, and money conducting research on effects that don’t actually exist
  
- Hard to exercise false positives from the literature

- Not enough incentives for researchers to conduct replications that debunk the false positives

- Erodes the credibility of psychological science
---
class: inverse center middle

# Why do we have a problem?
---
# Researcher Degrees of Freedom

- Researchers may engage in questionable research practices (QPRs) not because they intend to be dishonest.
Rather, they are motivated to make decisions that support their hypotheses

- Make decisions so that p-value will be below .05 and can claim statistical significance

<img src="fork.png" width="50%" style="display: block; margin: auto;" />
---

# Researcher Degrees of Freedom

- Which DV do you focus on?
- Measure the mean or the median?
- Which participants do you exclude?
- When do you stop collecting data?
- What analyses do you do?
- What covariates do you include?

**Making each decision increases the false positive rate (Type 1 error)**
---
# Covariates

> Covariate: a variable in a statistical analysis that was typically not controlled or manipulated during data collection but was measured

- Listening effort and pupil size 
  - covariate: Time
---
# Researcher Degrees of Freedom

---

# How many ways are there to analyze this data?

---

- Include people who dropped out
- Exclude people who dropped out
- Had to change plans
- Self-reported improvement
- Moderate headaches
- Severe headaches
- Under 50 
- Over 50 
- number of days with pain
- Average pain rating, all days
- Average pain rating, headaches
- Self-reported improvement
- Certain medications
- Stress 
- Hormonal
---
# Where do false p values come from?

> P-hacking: trying lots of analyses until you get a desired outcome

---
# QPRs: P-hacking

---
# Check this out

---
# QPRs: HARKing

> Hypothesizing After Results are Known (HARKing): Presenting exploratory findings as confirmatory

---
# QPRs: HARKing

> Hypothesizing After Results are Known (HARKing): Presenting exploratory findings as confirmatory

<img src="harkingscimethod.png" width="70%" style="display: block; margin: auto;" />
???
harking is bad because never can be falsified.

---

# QRP: Optional Stopping

- Data peeking

- Run some participants, look at data (stop if desired threshold is reached)
  
- **do not do this unless you correct for it**

- Inflates Type 1 error

---
# Peeking Simulations
<iframe src="https://shiny.psy.gla.ac.uk/debruine/peek/" width="100%" height="400px" data-external="1"></iframe>

---
# Publication bias

- Traditionally, journals were biased in favor of publishing:

1. Significant findings

- Non-significant findings relegated to a researcher's “file drawer”
    
2. Novel findings

- Counterintuitive, surprising findings more likely to be published in top journals
---

# Interim Summary

1. Researcher degrees of freedom lead to flexibility in how we analyze our results

2. Publication bias makes it hard to get findings that are not statistically significant published

3. Scientists are incentivized to publish

---

# Does this really mean we are f**ked?

---
<img src="jamesh.png" width="70%" style="display: block; margin: auto;" />

---
# Credibility Revolution

---
# Potential Solutions

---
# 21 word solution

“We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”

---
# Potential Solutions

- Preregistration

- A formalized (time-stamped) document that specifies all hypotheses & methodological choices in writing prior to data collection

- Reduces RDoF
  
  - Can’t p-hack
  
  - Can’t HARK
]

---
# Potential Solutions

- Preregistration
  
    - OSF
    
    - Aspredicted.org
]

]
---
# Potential Solutions

- Registered Reports

- Review and acceptance prior to data collection

<img src="rrr.png" width="80%" style="display: block; margin: auto;" />
---
# Potential Solutions

- Reduces RDoF
      
      - Can’t p-hack
      
      - Can’t HARK
      
      - Reduces publication bias 
]

---
# Last Class
<iframe src="https://app.sli.do/event/oudumzcz" height="50%" width="50%" frameBorder="0" style="min-height: 560px;" title="Slido"></iframe>

---
# Potential Solutions

- Sharing

- Posting all data and code on OSF
  
  - Sharing stimulus materials, lists, etc
---
# Potential Solutions

- Increased power/increased sample size

- Effect sizes in psychology are **smallish** 
  
> Small effect sizes can only be accurately detected with high statistical power

- Simmons et al. (2011) suggest at least N = 20 per group, but this isn’t large enough
  
- Use of within-participants designs affords more power

- More online studies?
---
# Open Science

- What is it?
  
> The movement to make scientific research, data, and dissemination accessible to all levels of an inquiring society

<img src="umbrella.png" width="80%" style="display: block; margin: auto;" />
]

.pull-right[
- Transparency 
  - Preregistration
      - **OSF**
      - Aspredicted.org
- Openness
      - Papers but also data
- Integrity
]
---
# Open Science Framework

> OSF is a free and open source project management tool that supports researchers throughout their entire project lifecycle.

<img src="osf.png" width="60%" style="display: block; margin: auto;" />
---

# ORCID

- Go to https://orcid.org/register

---
# Preregistration

1. Create an OSF account (with ORCID ID)

2. Start a preregistration
  -  First, sign in to the OSF, and go to https://osf.io/prereg/.
  
  -  You will be taken to the "OSF Preregistration" landing page.

<img src="step1.png" width="50%" style="display: block; margin: auto;" />
---

# Preregistration

- Click start a new preregistration

---
# Preregistration

- A textbox will appear

- Enter a title for your preregistration into the textbox, then click **Continue** 
<img src="step2.1.png" width="70%" style="display: block; margin: auto;" />

---
# Preregistration

- An OSF project will be created, and you will be taken to the project's **Registrations** page.
- Click the **New registration** button

---
# Preregistration

- Select OSF Preregistration from the list, then click the **Create Draft** button.

<img src="step4.png" width="70%" style="display: block; margin: auto;" />
---
# Preregistration

- The preregistration form will appear

---
# Preregistration

- The preregistration form will appear

<img src="step5.png" width="70%" style="display: block; margin: auto;" />
---