ManyClasses 1: Call for Applications

What is ManyClasses?

ManyClasses is a collaborative research project investigating the generalizability of educational practices across a variety of authentic college classes. Rather than conducting a research study in just one class, ManyClasses will examine the same research question in dozens of contexts. Learn more at https://manyclasses.org.

What is the purpose of this ManyClasses 1 study?

The purpose of this first ManyClasses study (ManyClasses 1) is to examine how the timing of feedback affects student learning in authentic college courses over the course of a single semester (Fall 2019).

Specifically, we will compare the effects of immediate feedback (i.e., feedback provided right after an assignment is submitted) versus delayed feedback (i.e., feedback provided several days after an assignment is submitted) on student learning outcomes. Additionally, given the fact that students automatically see immediate feedback but not delayed feedback, another factor we will examine is whether students are assigned to view the feedback provided.

We chose to investigate the timing of feedback in ManyClasses 1 for several reasons:

  • The provision of feedback is highly recommended for improving student learning (e.g., Benassi, Overson, & Hakala, 2014; Pashler et al., 2007)
  • The provision of feedback is a common component of most classes and will not represent a deviation from normal practice
  • Instructors must decide when to provide feedback to their students and they need evidence-based recommendations to inform that decision
  • There are recent claims that delayed feedback is more effective than immediate feedback (see Butler & Woodward, 2018), which contrasts with current recommendations to educators to provide feedback immediately after student work is submitted

Who is eligible to apply?

College instructors who are teaching an undergraduate credit-bearing course in the Fall 2019 semester that:

  • Uses Canvas as the online learning management system
  • Includes or could be adapted to include at least two Canvas quiz assignments on different topics and that are automatically scored
  • Includes or could be adapted to include a measure of student learning that is administered after the Canvas quiz assignments and that independently assesses content from the assignments using different items
  • Has a projected enrollment of at least 20 undergraduate students

We are interested to include a wide range of different classes, topics, formats, disciplines, and student populations. Thus, we encourage applications from many different types of instructors who teach many different types of classes. Instructors do not need prior research experience nor familiarity with educational research to apply. If you have questions about whether a class meets these criteria, please don’t hesitate to email us at contact@manyclasses.org.

What is expected of selected instructors?

The primary responsibility of the selected instructor is to facilitate the recruitment of students for ManyClasses 1 by announcing the research opportunity to one course that meets the eligibility requirements. Prior to or during the Fall 2019 semester, instructors will also be asked to:

  • Add an approved researcher to their Canvas course who will facilitate the implementation of the feedback conditions on selected assignments
  • Adding a researcher to a Canvas site will require institutional approval, and we will work with selected instructors to obtain this approval.
  • Include an approved statement on the course syllabus that informs the students that there will be minor variations in course materials
  • Provide the research team with the Canvas quiz assignments
  • Provide the research team with the course assessments (e.g., exams) and a mapping of which assessment items assess content from any given Canvas quiz assignment
  • Be willing to assign students to view the provided feedback and add some value (e.g., class points, extra credit, etc) to these assignments

What are the potential benefits for selected instructors?

There are at least three potential benefits for college instructors who apply and are selected to be involved in the first ManyClasses research project:

  • Participation in an ambitious research project that seeks to bring researchers and educators together to better understand the generalizability of educational practices across a variety of authentic college classes
  • Access to direct empirical findings on the effects of immediate versus delayed feedback on course assignments in the instructor’s class
  • Authorship on subsequent publication of the findings of ManyClasses 1, pending article acceptance

What is expected of students in participating classes?

Across the semester, all students will engage with assigned course materials as they normally would. Students will be invited to consent to their data being included in the ManyClasses study. Consenting to participate in the research study does not change what students will do in the course, it will only change whether researchers may analyze their student data.

On any given assignment, there will be slight variations in the timing of the feedback, and, across classes, there will be variations in whether students are assigned to view that feedback. However, within a class, all students will be exposed to the same variations. For example, one student may receive immediate feedback on the first Canvas quiz assignment and delayed feedback on the second Canvas quiz assignment, and another student may receive the reverse.

At the beginning of the semester, students will be informed of the opportunity for their coursework to be included in this research project. Prior to completing the first Canvas quiz assignment, they will complete a consent form and a FERPA waiver, which allows them to opt-in or opt-out of the study. For students who opt-in, their data from the course will be shared with the research team. For students who opt-out, they will engage in the same course activities, but their data from the course will not be shared with the research team.

How will participating classes be selected?

As mentioned above, our goal is to sample a wide range of different classes, topics, formats, disciplines, and student populations. While we certainly invite participation from instructors who teach classes about experimental research, cognitive psychology, educational psychology, or other topics related to this study, we plan to select our sample so that it represents the diversity of college courses. In addition to applying, you can also help by spreading the word to your colleagues in other disciplines!

While we would like to include the participation of all applicants, this first ManyClasses study is going to involve a great deal of manual effort on behalf of the research team. Under our current IRB protocol, the instructors of participating classes cannot perform research-related tasks (such as randomly assigning students to treatment conditions, differentiating assignments to different conditions, aggregating research data, etc.), and there is no software available for automating these tasks in live courses. For these reasons, a researcher will need to manually implement the study in participating classes, and this researcher’s time is limited. In the future, we hope to develop software that will facilitate much more inclusive ManyClasses studies with no practical limit on the number of participating classes.

Who is running this project?

The ManyClasses team is:

  • Paulo Carvalho (Carnegie Mellon University)
  • Josh de Leeuw (Vassar College)
  • Emily Fyfe (Indiana University)
  • Rob Goldstone (Indiana University)
  • Ben Motz (Indiana University)

This research study is being managed by research personnel from Indiana University and it has been approved by the IRB (Protocol #1812689804).

Who do I contact if I have questions?

If you have any questions or would like further information, please email the research team at contact@manyclasses.org. You can also learn more at https://manyclasses.org