Empirical Research Methods in Software Engineering and Informatics

Today, more and more papers get published that contain an element of empirical research methods in them. In fact, empirical methods are so prominent now, that for instance the VL/HCC conference demands that “Research papers are expected to support their claims with appropriate evidence." follwed by a reminder that there are also other ways to provide evidence for claims besides human subject studies. Similarly, the MODELS conference series defines one of three categories of research papers as “Papers evaluating existing problem cases or scientifically validating proposed solutions through, for example, empirical studies, experiments, case studies, simulations, formal analyses, and mathematical proofs.” So, clearly, empirical methods are considered important today.
However, they are not (yet) widely taught to our graduate students. Therefore, the course „Empirical Research Methods in Informatics“ was introduced at the Technical University of Denmark (DTU).

Goals

The course focuses on the practical aspects of empirical research, that is, there will be only short introductory lectures on the various topics. After that, participants shall work hands-on with real case studies (some of which derive from my own work).

For instance, participants will

• study existing experimental setups with a view to finding shortcomings and threats to validity and how they impact the claims put forward in a given article;
• develop a small experimental setup for a given case study in small groups and present their design in a plenary discussion; and
• define and refine a research question, and discuss alternative approaches to providing evidence for or against it.

Of course, there is more to empirical research than what can be taught in a tutorial. Thus, the objective of this course is only to provide its attendees with a starting point, possibly removing any inhibitions that might be there, and equip aspiring researches with some skills and first practical exercises in leading methodologically sound research.

After attending this course, participants

• are aware of the potential and limitations of empirical research methods;
• are capable of choosing an appropriate research paradigm for a given problem; and
• can assess the quality of the empirical research reported in an article, such as for a review.

Given the time restriction, we will focus mostly on controlled experiments as far as the practical exercises are concerned. Other paradigms will only be covered by lecture-format introductions.

Target Audience

The course is intended for advanced MSc-students and beginning PhD-students with little or no background in empirical research methods, but that realize that this topic is relevant for their studies. We will assume general knowledge of Software Engineering for the case studies.

Installments

• Tutorial at VL/HCC 2012 (cancelled)
• Tutorial at ECOOP 2013 (results)
• Summer School at DTU 2013 (results)
• Spring School at Universita di Trento, May 2014
• Summer School at DTU 2014
• Summer School at DTU 2015
• Summer School at DTU 2016

Lecturer

Harald Störrle has received a Dipl.-Inform. and a Dr. rer. nat. from the Universities of Hamburg (1997) and Munich (2000), respectively.

From 2001 to 2009 he worked as a software architect and methodology consultant in industry and as an adjunct lecturer at Munich University.

Moving back to academia in 2006, he held lecturer positions at the Universities of Innsbruck and Munich, is currently Associate Professor at the Danish Technical University (DTU) in Lyngby near Copenhagen.

He is a member of GI and ACM, a Past Vice President of the German Chapter of the ACM, and a current board member of the ACM Europe Council. His professional interests include Modeling and MBSE, Requirements Engineering, software development processes and methods, human computer interaction, and, of course, empirical research methods.

  Top