The last 5 years has led to a growth in human-robot interaction across all levels of AI, robotics, and cognitive science. The focus in this emerging field is how people interact with the robot itself. The ability to present sound results for HRI frequently requires that systems be evaluated with an appropriate experiment. Designing experiments has traditionally been conducted those with training in Psychology and/or Human Factors. As engineers and roboticists become involved in the development of systems requiring evaluation, it is important that these individuals conduct sound user evaluations that provide valid results.
The design of an experiment is critical to obtaining the required results. The initial step is determining what research question to answer. The proper definition of the research question drives the design of the experiment. Once the question is defined, a number of experimental factors are defined such as what is manipulated, what is measured, predictions, the basic design (e.g. within or between subjects), the number of participants, the characteristics the participants should possess, analysis methods, etc.
One critical component of designing and experiment within HRI is the decision to use simulated or real robots. This is a critical question that can affect the generalizability of the results: the pluses and minuses of using high-fidelity simulations vs. actual physical robots must be taken into account. When conducting user evaluations with real robots, particularly groups of robots, there are many factors to consider and the experimenter must be prepared to handle them. Our tutorial will address these topics and many more, including:
Creating a research question (much trickier than most people think).
Determining whether to use simulated robots or real robots (with a focus on what types of questions can be answered with simulations).
Determining what will be manipulated (with a focus on internal and external validity and types of designs).
Determining when a field experiment vs. a controlled laboratory evaluation is appropriate.
Determining what will be measured (basics like reaction time and accuracy as well as more complicated measures like eye-tracking and protocol analysis).
Determining what the predictions are (focused on theory-based predictions)
Determining who the subjects will be (ease of recruitment vs. ecological validity).
Determining how to analyze the experiment (not an in-depth tutorial on statistics, but focused on exploratory data analysis and very simple statistics like t-tests).
Determining the practical evaluation considerations when using real robots (focused on the common control problems that can introduce evaluation confounds).
Intelligent Systems Section
Naval Research Laboratory
Washington, DC USA
E-mail: trafton 'at' itd.nrl.navy.mil
Greg Trafton is section head of the Intelligent Systems Section at the
Naval Research Laboratory in Washington, DC. He is a cognitive
scientist with interests in HRI, interruptions/resumptions,
and the cognition of complex visualizations. Greg received his BS in
computer science (second major in psychology) from Trinity University
and his Ph.D in cognitive psychology from Princeton University.
Electrical Engineering and Computer Science Department
Vanderbilt University
Nashville, TN USA
E-mail: julie.a.adams 'at' vanderbilt.edu
Julie A. Adams is an Assistant Professor of Computer Science
and Computer Engineering in the Electrical Engineering and Computer Science
department at Vanderbilt University.
She conducts research in human-robotic interaction and distributed algorithms
for multiple robotic systems. She previously worked in Human Factors for
Honeywell, Inc. and the Eastman Kodak Company. Julie received her BS in Computer
Science and BBA in Accounting from