36 days ago, we enthusiastically announced that the basic research protocol for the ManyClasses1 experiment had been approved at IU. Since then, we’ve been navigating a difficult path toward adding additional institutions as data collection sites in this protocol. Our primary goal is that ManyClasses would be distributed across diverse educational contexts, and that means running the experiment in multiple institutions. This update discusses some of the challenges of distributing an experiment in classrooms across institutional boundaries, and how we’ve designed the ManyClasses1 study to address these challenges.
You know what would’ve made this relatively easy? A custom software tool for conducting this study. If we had a flexible educational assessment platform that could support experimental templates, could do random assignment in a crossover design automatically, could record granular activity logs, could send grades back to the LMS gradebook, and could filter non-consenting participants from data exports, we could just plug the tool into teachers’ Canvas sites, and the experiment would run itself. Alas, no one’s got that magic solution (we’ve looked! ASSISTments comes close – high-five to Neil Heffernan – but not all the way).
Lacking a software solution, we’re going to be running the experiment manually. Specifically, someone will need to randomly assign students to different groups manually (one-by-one), assign these groups to different versions of Canvas assignments, implement an informed consent statement in Canvas, and filter non-consenting students during data exports. These are all very “research-ey” tasks, and the person who performs these tasks needs to be a responsible approved researcher on the IRB protocol. Therein lies a catch-22: The researcher who does these tasks cannot be the teacher (IU’s IRB dictates that the teacher cannot know who has consented and who hasn’t, as a check against coercion, not to mention that many eligible teachers will not be experienced behavioral researchers or adept Canvas power-users); The researcher needs to access personally-identifying information about class enrollment all students (including those who have not consented – so that they can be filtered from the data exports), which is protected (in the USA) by FERPA.
It is possible for an institution to provide an exception to FERPA in accordance with 34 CFR § 99.31(a)(6)(i)(C), “to organizations conducting studies for, or on behalf of, educational agencies or institutions to … (C) improve instruction.” According to [Department of Education guidelines](https://studentprivacy.ed.gov/sites/default/files/resource_document/file/FERPA Exceptions_HANDOUT_horizontal_0_0.pdf), this exception clearly needs to come from the institution.
Thus, for a teacher to join ManyClasses, triggering the non-consensual disclosure of personally-identifying enrollment records to a researcher, we need a letter from a signatory official at the teacher’s institution – similar to how a research study conducted in grade school would need approval from the Principal. In our case, the signatory official needs to be a Data Privacy Officer or Registrar, and the letter also needs to name an executive sponsor who has legitimate interest in the results of the ManyClasses study.
Getting this letter signed by a Data Privacy Officer at a large institution is no small task, and we regret that this legal requirement will significantly narrow the range of classes that can participate in ManyClasses. We believe that this requirement may also limit the current study to institutions based in the United States, although we are not sure at this time. Nevertheless, we acknowledge the need to obtain administrative approval for a teacher to participate in ManyClasses, considering the sensitivity of student data.
To ensure a minimum range of participation, we are currently working to obtain letters from institutions that are members of the Unizin consortium. We will also work with teachers who apply to join ManyClasses to obtain these letters from their institution’s leadership. The approved IRB protocol and a template of the required letter are available on the ManyClasses1 OSF page: https://osf.io/q84t7/files/
So the bad news: Our ambition of maximizing the diversity of participation in ManyClasses is being restricted – only to those classes whose institutions are able to provide signed letters of support.
But the good news: The study is proceeding! We are in the process of securing a letter from the first institution outside of IU, and will be ramping-up recruitment in the very near future.
We sincerely appreciate your interest and support. If you have any questions, don’t hesitate to email us at: contact@manyclasses.org