Behavioral Software Survey Results

In Summer, 2014, we solicited participation in a survey on software for conducting behavioral experiments.  With some delay (due to school-year busy-ness), here is a summary of everyone's responses.  (We analyzed the data from responses collected through October, 2014.)

I'd like to thank Brendan Tomoschuk, graduate student here at UC San Diego, for help implementing and analyzing the survey responses.

You can download the raw data for everyone's response here.

Through October, 2014, we received 321 responses to our request (compared to 187 when a similar survey was conducted in 2006).  E-Prime remains the most popular software package among our respondents, with DMDX, Paradigm, PsychToolBox, Presentation, PsychoPy, and Experiment Builder rounding out the top 7.  These are the packages largely used for in-person experiments; for online experiments, the three most common packages were Paradigm, Javascript/PHP, and Ibex Farm.  Most of the top 10 packages were rated as relatively easy to use (between 5 and 6 on a 7-point scale), though Linger was especially easy and PsychToolBox and Javascript/PHP more difficult.  Ratings for the easiest packages for novices were Paradigm and Linger; PsychToolBox and JavaScript/PHP were rated as difficult for novices.  Most satisfied with their software were users of Paradigm, Linger, PsychoPy, and Experiment Builder; least satisfied were E-Prime, PsyScope, and PsychToolBox.  A significant proportion of responders for each package were looking to switch to something new only for E-Prime, PsychToolBox, and PsyScope; others were mostly sticking with the packages they're using.  Only E-Prime had a significant proportion of its respondents say they would not recommend their package to someone.

Some specific response patterns:

First, we asked which software package you used.  A summary of all responses follows, with the software package, the number of respondents who use that package, and the percentage of respondents who use that package:

E-Prime, 98, 30.5%
DMDX, 26, 8.1%
Paradigm, 26, 8.1%
PsychToolbox (MATLAB), 23, 7.2%
Presentation, 22, 6.9%
PsychoPy (Python), 21, 6.5%
Experiment Builder, 16, 5.0%
Linger, 11, 3.4%
JavaScript/PHP, 9, 2.8%
PsyScope, 9, 2.8%
OpenSesame (Python), 8, 2.5%
SuperLab, 8, 2.5%
MATLAB, 5, 1.6%
Ibex Farm, 4, 1.2%
Direct-RT and MediaLab by Empirisoft, 3, 0.9%
Python, 3, 0.9%
Dialogue Experimental Toolkit (DiET), 2, 0.6%
experigen, 2, 0.6%
EyeScript (Python), 2, 0.6%
EyeTrack (UMass), 2, 0.6%
Inquisit, 2, 0.6%
Qualtrics, 2, 0.6%
Adobe Flash (Action Script 3.0), 1, 0.3%
Ductus, 1, 0.3%
EEGlab in matlab, 1, 0.3%
Habit, 1, 0.3%
Mechanical Turk, 1, 0.3%
MEL, 1, 0.3%
NESU, 1, 0.3%
OnExp (PHP), 1, 0.3%
PsychToolbox (but with Octave), 1, 0.3%
Psyscript, 1, 0.3%
R, 1, 0.3%
StimPres, 1, 0.3%
Tobii Studio, 2, 0.6%
WebExp, 1, 0.3%
WebExp2, 1, 0.3%
ZEP, 1, 0.3%

Next, we asked what operating system you used.  Responses for the top 10 packages were:

E-Prime: 97 Windows, 1 Mac
DMDX: 26 Windows
Paradigm: 17 Windows, 9 iOS
PsychToolbox (MATLAB): 12 Mac, 8 Windows, 2 Linux, 1 Mac and Windows
Presentation: 22 Windows
PsychoPy (Python): 8 Mac, 8 Windows, 3 Linux, 1 Mac and Windows, 1 Windows and Linux
Experiment Builder: 16 Windows
Linger: 6 Mac, 3 Windows, 2 Linux
Javascript/PHP: 3 Mac, 3 Windows, 2 Linux, 1 Windows and Mac
PsyScope: 9 Mac

We asked whether the package was used for in-person experiments or remote/online experiments.  The top 10 packages used in-person were:

E-Prime, 97
DMDX, 23
PsychToolbox (MATLAB), 23
Presentation, 22
PsychoPy (Python), 21
Experiment Builder, 16
Linger, 10
PsyScope, 9
OpenSesame (Python), 8
Paradigm, 8

The top 8 packages used online were (with remaining packages used just once):

Paradigm, 15
JavaScript/PHP, 9
Ibex Farm, 4
DMDX, 3
Dialogue Experimental Toolkit (DiET), 2
experigen, 2
Inquisit, 2
Qualtrics, 2

We also asked how easy it is to build experiments in each package (1 = hard, 7 = easy).  The top 10 responses were (package, number of responses, average rating):

E-Prime, 98, 5.06
DMDX, 26, 5.46
Paradigm, 26, 5.88
PsychToolbox (MATLAB), 23, 4.52
Presentation, 22, 5.36
PsychoPy (Python), 21, 5.62
Experiment Builder, 16, 5.44
Linger, 11, 6.36
JavaScript/PHP, 9, 4.56
PsyScope, 9, 5.78

For how easy it is for a novice to build experiments, the responses were:

E-Prime, 98, 3.49
DMDX, 26, 3.62
Paradigm, 26, 5.46
PsychToolbox (MATLAB), 23, 1.96
Presentation, 22, 3.36
PsychoPy (Python), 21, 4.14
Experiment Builder, 16, 3.50
Linger, 11, 5.45
JavaScript/PHP, 9, 2.44
PsyScope, 9, 4.00

How satisfied are you with you package (1 = not at all, 7 = totally):

E-Prime, 98, 4.14
DMDX, 26, 5.04
Paradigm, 26, 5.77
PsychToolbox (MATLAB), 23, 4.78
Presentation, 22, 4.95
PsychoPy (Python), 21, 5.48
Experiment Builder, 16, 5.44
Linger, 11, 5.55
JavaScript/PHP, 9, 4.89
PsyScope, 9, 4.56

We asked whether you were sticking with your current package, looking to change, or don't know.  For the top 10 response-getters, people's responses were (package, total responses, sticking with, looking to change, don't know):

E-Prime, 98, 50, 34, 14
DMDX, 26, 11, 5, 10
Paradigm, 26, 21, 0, 5
PsychToolbox (MATLAB), 23, 14, 6, 3
Presentation, 22, 12, 4, 6
PsychoPy (Python), 21, 16, 0, 5
Experiment Builder, 16, 13, 2, 1
Linger, 11, 8, 1, 2
JavaScript/PHP, 9, 7, 0, 2
PsyScope, 9, 4, 5, 0


Would you recommend your current software package (package, N, yes, no)?  

E-Prime, 98, 69, 29
DMDX, 26, 21, 5
Paradigm, 26, 26, 0
PsychToolbox (MATLAB), 23, 18, 5
Presentation, 22, 17, 5
PsychoPy (Python), 21, 20, 1
Experiment Builder, 16, 15, 1
Linger, 11, 11, 0
JavaScript/PHP, 9, 7, 2
PsyScope, 9, 7, 2

We asked for general comments about what people liked about their software.  Roughly, these were:

E-Prime: easy to use, wide user base, flexible, powerful
DMDX: Free, flexible, precise
Paradigm: Inexpensive, easy to use, fantastic support
PsychToolbox: Flexible, easy to control
Presentation: Flexible
PsychoPy (Python): Open-source, easy
Experiment builder: great support, great integration with eye trackers
Linger: simple, free, does what it does well
JavaScript/PHP:  can do whatever you want with it, rules the internet
PsyScope: free, easy to use

We asked about what you didn't like about your package.  Roughly, these were:

E-Prime: Expensive, can be difficult
DMDX: Not very easy, only on windows
Paradigm: Want a Mac version
PsychToolBox:  Expensive, steep learning curve
Presentation: Expensive, limited freedom
PsychoPy (Python): a range of specific concerns
Experiment builder: Hard to learn, only for eye-tracking experiments
Linger: Has some quirks, hard (or impossible?) to do experiments online
JavaScript/PHP: steep learning curve
PsyScope: days are limited, button boxes not available, various

We would like to thank everyone who participated in this survey, and we hope the results prove to be useful.

Sincerely,

Vic Ferreira
Brendan Tomoschuk

Comments