Chapter 3 Major Project ONE (2 labs)

3.1 Overview

In this project you will attempt to replicate a recent short report published in Psychological Science. A replication of the experiment has already been designed. To collect data, you will first participate as a subject in the experiment. Then, as a class you will be introduced to the published paper. You will read the paper and it in class. Then, the class will analyze the collected data to determine whether or not the major effects of interest have been replicated. Data will be collected using pen and paper methods, and analyzed by computer software. Each student will write a 5+ page, APA style report on the project.

3.2 Things you will learn

  • Reading and citing primary source material

  • Writing a brief APA style research report

  • Conducting and reporting T-tests

3.3 Background readings

To be provided by Lab Instructor

3.4 Grade

  • 10% of final grade

  • Graded by your lab instructor. Lab instructor sets due date, and determines whether revised drafts are submittable.

  • If you submit completed versions of all paper assignments (1, 2, and 3), then you get to drop your lowest paper grade from paper 1 or 2. Specifically, you will receive your highest grade from paper 1 or 2 for both papers, thereby eliminating the lower grade. This allows room to learn and improve as you go.

3.5 Writing the paper

There are many resources for help on writing an APA style research report in the lab manual, on the textbook, and the website. Check them out. As well, here is a rough roadmap for writing paper 1.

3.5.1 APA formatting, Title and Abstract

  • Use APA formatting rules.

  • Create a suitable title for the paper

  • Write the abstract : No more than 250 words. The aim is to briefly describe the issue at hand, the experiment, and the results.

3.5.2 Introduction (around 2 double-spaced pages)

The goal of the introduction is to first put the research into a broader context, and then narrow the focus to describe the specific research aims.

  1. A. Opening section: (starting broad)

    • about 1 paragraph

    • Discuss a real-world example of the general phenomena under investigation by the paper

    • Tell the reader that the purpose of the current experiment is to conduct a replication of the work in question

    • Establish the domain and big questions.

  2. Middle section: Prior work

    • Discuss some examples of previous research that are similar to the present research. You have an opportunity here to look this kind of research up on Google Scholar. One or two examples ought to be enough.

    • Explain the specific question that is being asked in this replication work.

  3. Final section: (briefly explain the present aims, the experiment and what you expect to find)

    • Explain the hypotheses (alternatives)

    • Explain the logic of how the hypotheses will be tested

    • Briefly explain what the participants will be doing in the task

    • Briefly give predictions for performance in each condition

3.5.3 Methods (about 1 page)

The methods section should be a complete recipe that anyone could follow to replicate your experiment. There are lots of details that you can include, some of these are listed below. Be brief and concise

  1. Participants

    • how many people?

    • where did they come from?

  2. Materials

    • what were the stimuli?

    • how were they organized?

  3. Design & Procedure

    • What was the design

    • What were the independent variable(s)

    • What was the dependent variable

    • Within or between subjects?

    • How were the stimuli for each trial chosen

    • Describe the steps each participant took to complete the experiment

3.5.4 Results

The result section is used to report the patterns in the data, and the statistical support for those patterns. You will compute the results using SPSS in the lab computers. The lab manual can be consulted for help on running statistical tests, and for reporting results.

  • Describe the statistical analysis

  • Tell the reader where they can see the data. E.g., the results of experiment 1 are presented in table 1, or in figure 1

  • Make a table or figure to display the data in your paper

  • Report the statistical test, and the pattern of the means.

3.5.5 Discussion

The discussion can be used to briefly restate verbally the pattern of the most important results, and then to relate the results to theory and ideas developed in the introduction

  • Highlight the main findings from the experiment

  • Discuss how the data can be explained by the hypothesis. What inferences do you make about the hypotheses based on the research findings?

  • Broaden your discussion. Can the findings be explained by an alternative theory? What can be generalized to the real world? Are there important confounds that prevent us from interpreting our results?

3.5.6 References and Figures or Tables

  • Include citations used in the paper using APA style format

  • Include a figure or table to show the results

3.6 Data-analysis tips

In lab one you will be collecting measurements on several dependent variables, in each of two manipulated conditions (the independent variable). For each dependent variable you will want to determine whether the manipulation had an effect. That is, did the independent variable cause a change in the dependent variable. We know that differences can sometimes be observed by chance alone, so we want to conduct an inferential statistical test to determine the probability that our observed difference could have been produced by chance alone. To do this we will be conducting several t-tests. This is a short primer on the process. You can conduct t-tests in the software of your choice, or by hand using a calculator (or in excel). Here, we will use the free and open-source statistical package called R, to illustrate the process.

You can also look at the examples of conducting t-tests in software using the lab manual from PSYC 3400 Introductory Statistics, and refresh your understanding of t-tests by reading the t-test chapter from the PSYC 3400 textbook.

Let’s imagine we have two groups of 10 subjects each. Group A receives condition 1 of the independent variable, and Group B recieves condition 2 of the independent variable. We then measure some behavior for all of the subject in all of the groups. To make this more concrete, let’s say 10 subjects drink coffee, and the the 10 subjects drink tea. Then we present all of the subjects with a piece of art and ask them rate how beautiful they think it is on a scale from 1 to 7.

When we collect all the data we should have 20 total ratings, one for each subject in each group.

For example, if you put the data in a table it might look something like the following. Note, the grey text box shows the R code used to simulate the data. For, each group, we sample 10 numbers from a normal distribution with a mean of 4, and a standard deviation of .5. Then we put the numbers in a table.

coffee<-round(rnorm(10,4,.5))
tea<-round(rnorm(10,4,.5))
all_data<-data.frame(coffee,tea)
knitr::kable(all_data)
coffee tea
4 4
4 4
4 4
4 4
4 3
4 4
4 3
4 6
4 5
4 5

We can do some quick descriptive statistics, for example, we might want to know the means of the beauty ratings for the coffee and tea groups.

mean(coffee)
## [1] 4
mean(tea)
## [1] 4.2

The means aren’t very different, and of course we should expect they should be similar. After all, we sampled these means from the exact same distribution. So, we should expect that on average, the means should both be close to 4. However, they won’t necessarilly be exactly 4, because of variability introduced by random sampling.

3.6.1 The t-test

What we want to do next is conduct an independent samples t-test. We want to determine whether any possible difference between the coffee and tea groups could have been produced by chance alone. We can conduct a t-test in R very easily using the t.test function.

t.test(coffee,tea,var.equal=TRUE)
## 
##  Two Sample t-test
## 
## data:  coffee and tea
## t = -0.68825, df = 18, p-value = 0.5001
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.8105138  0.4105138
## sample estimates:
## mean of x mean of y 
##       4.0       4.2

R gives us back the t values, the degrees of freedom (df), and the associated p-value. The p-value tells us the likelihood that our difference, or a difference greater than the one we observed could have been produced by chance.

3.6.2 One more time

Let’s try this whole process again, but this time we will simulate data with an actual difference between the groups. For example, let’s say we want to simulate the idea that drinking coffee makes people think the art is less beautiful by at least 2 points, and then reconduct the t-test with the new simulated data. We will sample numbers from a normal distribution with mean 3 for the coffee group, and mean 5 for the tea group (for an average expected difference of 2).

coffee<-round(rnorm(10,3,.5))
tea<-round(rnorm(10,5,.5))
all_data<-data.frame(coffee,tea)
knitr::kable(all_data)
coffee tea
4 5
3 5
3 6
2 5
3 5
3 4
3 5
3 5
2 5
4 5 
mean(coffee)
## [1] 3
mean(tea)
## [1] 5
t.test(coffee,tea,var.equal=TRUE)
## 
##  Two Sample t-test
## 
## data:  coffee and tea
## t = -7.746, df = 18, p-value = 3.868e-07
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2.542456 -1.457544
## sample estimates:
## mean of x mean of y 
##         3         5

3.7 Writing up the results of a t-test

We’ve now conducted two different t-tests, and received different results on each them. You will likely find different results for all of the t-tests that you conduct for the lab experiment. However, you will use the basic sentence structure to report all of the results. When you report the results of your experiment along with statistical tests there are two important features to include, the pattern of the results, and the inferential statistic. In this situation, we would simply report the means and the t-test information. Here are is an with made-up numbers.

The coffee group gave a lower mean beauty rating (M = 3.4) than the tea group (M = 5.6), and the difference was significant, t (18) = 5.4, p < .001.

So, just in one sentence we tell the reader what the means were in both conditions, as whether the result was significant. APA style recommends reporting exact p-values when they are greater than .001 (for example p = .047). If the p-value is less than .001, then you just need to report p < .001.