Research Methods in Cognitive Science

class: center, middle, inverse, title-slide

# Research Methods in Cognitive Science
## Week 5: Experiment Basics
### Jason Geller, Ph.D.
### 2021-09-30

---

# Housekeeping

- Tuesday: Teams 1 and 2 meet me in A120

- Thursday: Sarah Colby Virtual Talk

---
Last Class

---

# Today

- IV and DV

- Experimental Designs: Between vs. Within

- Confounds and Selection Effects

- Critiquing Experiments

- More threats to internal validity

???

In today's lecture, we're going to talk in more detail about experimental design.

First, we're going to talk about the different types of variables that we see in experiments. We'll consider between versus within subjects designs, which represent two different types of experiments.

We'll consider two of the biggest threats to internal validity confound and selection effects.

we'll consider how to best critique the results of experiments in terms of the four validities

and finally we will consider more threats to internal validity

---
# Experiments

> Primary goal of experiments is to identify causal relationships between things in the world

- Two pillars of experimental design:

1. Manipulation

2. Random assignment

???
Experiments are the gold standard in cognitive science. This is because they allow us to make causal claims. Experiments afford us the opportunity to say this thing causes this other thing. There are two important design features that allow us to do this. 
---
# Cause and Effect

- Cause
  - Independent variable (IV): Manipulated

- Effect
  - Dependent variable (DV): Measured

???
So experiments let us set up the situation where we can observe causal relationships. When we want to see a relationship between a cause and effect, we first manipulate one variable, and that variable is called the independent variable or IV.

11
00:02:10.800 --> 00:02:17.700
We then see what the consequence of that manipulation is on our dependent variable or DV.

12
00:02:18.990 --> 00:02:30.840
The dependent variable is measured or observed passively, and we see whether or not it changes as we control or manipulate the independent variable.

13
00:02:31.860 --> 00:02:42.180
So the independent variable or IV is our cause in our experiment, and our dependent variable or DV is our presumed effect.

14
00:02:48.000 --> 00:02:53.820
Alright, let's consider an example study to help us learn a little bit more about how experiments are constructed.

---
background-image: url(expt.png)
background-position: center
background-size: cover
---
background-image: url(threat1.png)
background-position: center
background-size: cover

---
background-image: url(ran.png)
background-position: center
background-size: cover
---

# Between vs. Within Designs

- Between-participants design

> Each participant experiences only one condition in the design. Chance determines which condition.

---
background-image: url(within.png)
background-position: center
background-size: cover
---
background-image: url(threat2.png)
background-position: center
background-size: cover

???

0:11:23.250 --> 00:11:33.630
So here, all participants type notes in the first week of the study, and all participants then write notes in the second week of the study.

64
00:11:35.820 --> 00:11:41.340
It could be that order of the treatments is explaining any results that we see.

65
00:11:42.630 --> 00:11:52.830
So maybe it's the case that just observing people over time, they tend to report more symptoms when we initially observed them, than when we later observe them.

66
00:11:53.370 --> 00:12:01.620
So, people might have more headaches in the first week of the study than they do in the second week of the study, just because they're kind of getting used to being observed.

67
00:12:04.740 --> 00:12:08.970
A solution to order effects is called counterbalancing.
---

background-image:url(cb.png)
background-position: center
background-size: cover

---
# Pros and Cons of Between and Within Designs

- Between-participant designs:

- Require larger sample sizes (sometimes 100s depending on effect size)
  
  - But, fewer worries about contamination across conditions

- Within-participant designs:

- Participants serve as their own controls (increases statistical power)
  
  - Worries: Order effects, experimental demand

---

background-image: url(confound.png)
background-position: center
background-size: cover

---

background-image: url(vitc.png)
background-position: center
background-size: cover
---

background-image: url(valid.png)
background-position: center
background-size: cover

---
# Review

- Experiments have a manipulated variable (IV) and a measured variable (DV)

- Ps experience just one condition (between-Ps) or all conditions (within-Ps)

- Watch out for order effects, confounds, and selection effects

- Use the four validities to critique experiments

---
# The bad experiment

- One-group, Pre-test/Post-test design

---
# Maturation

> Participants are changing naturally over time between  pre-test and post-test.

- Rule out by using a control/comparison group.

---

# History

> An event intervenes between pre-test and post-test to change participants

- Maybe something big happened in the news that was related to your study.

- Rule out by using a  control/comparison group.

---

# Regression to the mean

> Participants are selected for intervention based on their extreme scores. On re-measurement, their scores become more moderate

- A group's average is extremely depressed at pretest, in part because some members volunteered for therapy when they were feeling worse

- Rule out by using a control/comparison group.

---

# Attrition

> Participants drop out of the study at different rates fromtreatment group and control group

- Maybe they got:

- Illness
    - Died
    - Left the experiment

---
# Testing

> Participants’ responses change over repeated testing.

- GRE verbal scores improve only because students take the same version of the test both times

- Solved by using a between-subjects, post-test only design

---
# Instrumentation

> The meaning of a measuring instrument changes over repeated use. Changes are not due to the IV.

- Coders become bored or fatigued over time

- Solve by using masked coders, randomly assigning coders to stimuli,, or training

---
background-image: url(3morethreats.png)
background-position: center
background-size: cover
---
# Review

- All told, a dozen different threats conspire to mess up causal claims

- Be attentive to control/comparison group

- Watch out for differences between  treatment and control (beyond the one intended change)