Research Methods in Cognitive Science

class: center, middle, inverse, title-slide

# Research Methods in Cognitive Science
## Week 4: Replication…Replication…Replication
### Jason Geller, Ph.D.
### 2021-09-23

---

# Outline

- Types of replication

- Close
  - Conceptual

- Evaluating Replications

- The replication recipe (Brandt et al. 2014)

- Why teaching replication is important

- Replication Activity

---
# Why?

<img src="type12.png" width="30%" style="display: block; margin: auto;" />
- **Influential difference between the two experiments?**

---

<img src="cookiefail.png" width="70%" style="display: block; margin: auto;" />
---

# Types of Replication

- Direct/exact/close replication

- New experiment/study matches the original as closely as possible
  
- Conceptual replication

- New study attempts to test the same idea as the original, but may differ significantly in implementation
---
class: inverse center middle

Replications are not strictly "successes" or "failures"

---

- Not all studies provide equally strong evidence

.pull-left[

- Study 1

- N = 20, two groups
  - Not-preregistered
  - Reports novel, unexpected finding

]

.pull-right[
- Study 2 (close replication)
  - N = 200
  - Preregistered
  - Does not demonstrate novel, unexpected finding
]

---
<img src="doesit.png" width="100%" style="display: block; margin: auto;" />

---
background-image: url(guide.png)
background-position: center
background-size: cover
---
# Brandt et al. (2014)

- A 36-question guide to conducting convincing (good) replications

- 5 main ingredients

---
# Ingredient 1

- Carefully define the effects and methods that the researcher intends to replicate

---
# Ingredient 2

- Follow the methods of the original study as closely as possible

- Sometimes not possible
  
      - Authors do not provide enough detail
      - Authors do not share their materials
  
  - Contact them (for projects you may have to do this :))

---
# Ingredient 3

-  High statistical power

- Rule of thumb: collect 2.5 times the original sample size
  
     - This is what we are shooting for here

---
# Ingredient 4

- Make complete details about the replication publically available

- Publicly posting protocols, stimuli, and other materials on OSF

---

# Ingredient 5

- Evaluate replication results and compare them to the results of the original study

- Not enough to say success/failed based on significant result

- Note important methodological differences: 
  
    - Online vs. lab
    - Equipment (CRT vs. LCD)
    - Cultural differences
    
<img src="differences.png" width="70%" style="display: block; margin: auto;" />
---
# Teaching Replication Is Important

- Educational benefits:

- Fits the needs of the full range of students
 - Learn to read an article closely
 - Trace the steps of the experts
  - Gather relevant materials
  - Plan logistics of carrying out protocol
  - Create an analysis plan
  - Recruit participants and collect data
  - Analyze data and write up results
  - Compare results to original findings
- Scientific benefits:

- Addresses replication crisis
  - Employs best practices
---
# Replication Activity

.pull-left[

- Strak, Martin, & Stepper (1988): Facial Feedback Hypothesis

- Our facial expressions can trigger emotional reaction

.pull-right[

]

---
# Group Discussion

1. Do you think this study replicated?

2. Why/Why not?

3. How much are you willing to bet on it?

3. What are the potential issues with this study (i.e., has anything changed since this study?) Would findings replicate in different cultures?

---
# RRR (Wagenmakers et al. 2016)

- 17 labs with over 1000 participants from different countries

---
# Concluding Message

- Replication is important!

- Science is a continual process of updating what we know (self-correcting)

- Psychology has changed in the last decade

- Open science is becoming the norm (I hope)
  
- Science/Psychology/Cognitive Science is not fucked :)