Over the course of my work with adaptive tutors and MOOCs, I’ve had the pleasure of building randomized controlled experiments within several platforms, including Moodle, NovoEd, EdX, Khan Academy, and more recently ASSISTments. I opted to write this foreword to help external researchers seeking to create content-based randomized controlled trials or AB experiments in a relatively clear cut and uncomplicated manner. Many technology companies frequently alter their code to add ‘if, then’ statements that alter site behaviors for different users. However, the goal is not scientific publication, and the study and results are not easily attained.

When performing studies in many of these platforms, the researcher is responsible for altering the underlying source code and having a platform representative reboot the system to implement the experiment. These systems make it difficult for researchers to manipulate content and code, often requiring programming knowledge and a fair amount collaboration, thereby reducing the researcher’s ownership of the study and its outcome. They also impose a burden on the collaborating organization themselves.

The value of ASSISTments is that you can build your experiment directly into content, rather than altering code, thereby allowing any author to build and experiment that goes live almost instantly. This approach is made possible by the fact that researchers are not able to create experiments that would keep the system from working properly for teachers and students. The programmer is largely removed from the equation, leaving the researcher in charge to create their study using the available architecture. The source code is not changing. Collaboration still allows your study to be approved.

While this ‘experiment on demand’ capability is impressive, it is somewhat limiting in that the range of functionality for building an experiment. There is also a bit of a learning curve when it comes to understanding the architectural limitations of study designs within ASSISTments. However, there are several different types of experimental paradigms that are possible within this platform and the ASSISTments team is continually trying to expand architectural support for new types of studies.  Communicate with WPI your needs, as overtime, with support from NSF, ASSISTments will be trying to figure out the most useful next steps to add to the Testbed.

You can see an example of a study I am doing with WPI, examining motivation. 

If you are interested in doing experiments in online educational resources more broadly, you might explore resources at www.josephjaywilliams.com/experiments-online-education. These include symposia, workshop papers that act as how-to guides, examples of experimentscomparison of technology for in vivo experiments, and the MOOClet Framework for simultaneously experimenting and personalizing.

I’ve had the pleasure of contributing to portions of this user manual and I hope you enjoy trying to figure out how to build your experiment within this platform. 

Joseph Jay Williams, PhD

Research Fellow, HarvardX, Harvard University