What’s That, You Want to Run an Online Experiment?

A few months back, over at blog.prolific.ac, we published a post exploring the options open to researchers when recruiting participants for online research. 

In case you’ve been living in a cave for the last few years, online human data collection is now big business. Researchers from a range of fields (psychology, clinical research, population health, social sciences, economics and more) are turning to the web to supercharge their science. 

Compared to traditional sample collection methodologies (in-person, post, or phone) online participant recruitment offers several distinct advantages: Faster data collection, larger samples, reductions in costs, and importantly, more diverse populations. Arguably, online samples are more representative of the national population than a typical ‘WEIRD’ sample of university students.

This post will explore some of the tools and platforms that can help with a key stage of the online research process: creating your survey or experiment. Specifically, we’ll be looking at options for running online experiments, with a slight focus on the more complex platforms – those designed to collect reaction time data (e.g., cognitive tasks), or to deliver complex experimental paradigms with a range of response types. We’ll examine the pros and cons of Qualtrics, Gorilla, Inquisit Web, as well as the good old DIY approach. 

Before we get started, full disclaimer: This is a guest blog written by Prolific (a platform for online participant recruitment). We see a lot of online research and have a pretty good handle on the types of experimental software that researchers typically use. That said, if you think we’ve missed something, then please let us know. We’re scientists at heart and always keen to learn more! 

Qualtrics

Qualtrics is most famous for their survey software, which combines survey-creation tools with hosting and data storage. Qualtrics only supports survey-style data collection – free-text, multiple choice answers, rating scales, etc. However, their randomization tools allow researchers to build survey experiments; for example, by assigning participants different block of questions to answer, or different materials to read, watch, or listen to. 

Surveys are built in Qualtrics using a graphical interface, meaning it’s easy to get an experiment running, and that no programming knowledge is necessary. The ‘Survey Flow’ panel shows how participants will progress through the experiment, and allows you to control when allocation to condition occurs.

Ultimately, a major advantage of Qualtrics is that many universities and departments already have subscriptions, so you’re often surrounded by people who’ve used the platform before. It’s also a doddle to connect a Qualtrics experiment to Prolific, and you can even send participant IDs directly to Qualtrics from Prolific, meaning you’ll never lose track of a survey submission.

Qualtric’s biggest downside is its narrow focus. If you want to run anything more complex than a simple survey experiment (collect reaction time data, adapt your test to participant performance, allow participant interaction, or run a longitudinal experiment that saves data between sessions) then Qualtrics isn’t the platform you’re looking for. 

Gorilla

After Qualtrics, Gorilla is the second most popular platform on Prolific, and we can see why! Gorilla is a specialist online experiment hosting service that provides fully-featured tools for experiment design, in addition to data storage and hosting. 

As you might expect for an online-experiment platform, Gorilla supports complex experimental tasks such as N-back, the implicit association test (demo), prisoner’s dilemma (demo) and game-like tests such as Tetris and Tower of Hanoi. Multiple input methods (mouse, keyboard, mouse training, voice recording and basic eye-tracking) are available, and responses are recorded with millisecond accuracy, making it an appropriate platform for cognitive assessment. 

Experiment design in Gorilla is done through a graphical user interface, which separates the overall survey flow and randomization from the nitty-gritty details of any tasks you might use. It’s easy to add survey-style questions between more complex tasks, and visualize how participants will be allocated to condition. 

It’s worth saying that Gorilla has tonnes of features and options, meaning it can feel overwhelming to a new user. And, although you can use a graphical interface for most of your study design, once you get beyond the basics, Gorilla can get complex very quickly. That said, given the power that Gorilla offers, we think it’s worth wrestling the beast and conquering the learning curve. And at least integration with Prolific is easy…

Inquisit Web

Inquisit Web (by Millisecond) offers a similar, but more technical, alternative to Gorilla for building online experiments. Inquisit’s focus is on high precision, high control tasks, and allows the researcher to create a remote experience comparable to that in the lab. 

It’s both an advantage and a disadvantage that Inquisit Web uses downloaded software to run its experiments. On one hand, the need for a software download means Inquisit can ‘escape the browser’ and provide superior timing accuracy to that achievable in JavaScript. It also means researchers can be assured their task fills the whole of the participant’s screen, and that they’re not alt-tabbing every minute to check Facebook (We know they’d never!)

On the flip side, software downloads are a common gripe we hear from participants at Prolific (normally due to device incompatibility issues), so you may upset some of your potential sample! 

Experiments are built in Inquisit using a ‘streamlined scripting language based on JavaScript.’ There’s no graphical user interface, meaning you’ll need to do some coding yourself, but there is good documentation and an active support community. There’s a pretty steep learning curve, but if you’re after the most precise timing, then Inquisit Web is what you need.

Do it yourself

Sometimes the study you’ve got planned is just too unusual for any existing solutions. Maybe you need a real-money prediction market to explore to what extent a person’s prior beliefs affect the likelihood of them placing a bet? Maybe you want to sample thousands of Western speakers to understand how they pronounce Chinese place names? How about a collaborative pub quiz, with teams made up of other online players? Or a longitudinal investigation of whether a more ‘gamified’ task can reduce participant attrition?

We’ve seen plenty of experiments like these on Prolific, and all of them used bespoke software to achieve their aims. Collectively, DIY platforms are the second most popular approach to experiment building on Prolific. So, if you’re planning to develop your study from scratch, then you’re in good company.

Developing online experiments is obviously a huge topic, and we can’t explore it here fully. That said, here are some libraries which you might find inspiring:

• Nodegame is a JavaScript library which provides support for large-scale, real time, multiplayer experiments.
• lab.js is a lovely little library for building social science experiments. It includes a graphical experiment builder, but you’ll need a bit of technical know-how to get your study deployed online.
• JSPsych is a script-based library with a collection of psychological tasks. Or, if you’re familiar with PsychoPy, then PsychoJS is its browser based cousin.
• Finally, if you want to go the whole hog and build your experiment from scratch, then Firebase is possibly the simplest data storage and hosting platform on the web, and PixiJS provides solid functionality for precisely displaying graphics within the browser. 

The main advantage of a DIY experiment is complete flexibility – you can do anything your secret nerdy heart desires (and your programming skills allow). Cost is often a non-issue, since there are many free experimental libraries available, and data storage on a platform such as Firebase costs pennies for the scale of most research projects. 

There are however, several disadvantages. The skills required to program online experiments are neither quick nor easy to learn, and support comes only from limited documentation and community forums. Self-developing also means that practical details, such as server load and data security, become problems that you need to worry about. 

Thankfully, the easiest part of developing your own online experimental platform is the integration with Prolific (unexpected, right?). All you need to do is get the PROLIFIC_PID parameter from the URL. Job done. 

Conclusion

If after all that you still haven’t found a suitable platform for running your online experiment, it’s worth saying that we couldn’t cover everything. There are many (and more) options out there, and new ones are appearing all the time. 

Ultimately, choosing how you build your experiment involves compromise between the requirements of your study design, the type of data you intend to collect, and practical things such as technical expertise, cost, etc. 

If there’s one thing we’ve learned at Prolific, it’s that every research project has its own complexities, twists and turns. We hope this overview helps you find an experimental platform that suits you.

The Prolific Support Team is always on hand, whether you need guidance setting up your experiment or help integrating your experimental platform. Any questions, let us know @prolificac!