V+Ü Empirical Methods in Software Engineering (Empirische Methoden im Software Engineering, vormals bekannt als: Empirische Bewertung in der Informatik) SS 2022

Description

Software Engineering is a field of so-high socio-technical complexity that the properties (let alone the usefulness) of proposed methods and tools are not at all obvious. We need to evaluate them empirically.

This course introduces two different manners in which one can think about this situation and approach evaluations:

1. A quantitative perspective. This aims at quantified statements about the tools and methods and is based on a positivist epistemological stance and corresponding culture.
2. A qualitative perspective. This aims at making sense of the things that are going on to create the phenomena that give rise to the quantitative outcomes. This perspective is based on an interpretivist epistemological stance and has a culture that values different things.

Both perspectives have different strengths and weaknesses and are suitable for different types of research interest. In this course, we will learn to think in both of these perspectives and to appreciate the different benefits they provide. We will learn what it means that a study has high quality: it has high credibility and high relevance. We will train diagnosing the various quality problems that often reduce credibility or relevance.

We will work through the most common research methods and will discuss real examples (interesting published studies) of each, along with their strengths and weaknesses.

Participants will understand how and when to apply each method and for one of them develop some practical skills by doing so.

---

Administration

Lecturers

• Lutz Prechelt (lecture)
• Linus Ververs (tutorial)

Requirements/target group, classification, credit points etc.

see entry in the KVV course syllabus

Registration

Dates

• The lecture is held every Monday from 10:15 to 11:45 in room 049, Takustr. 9
  • The first lecture is on Monday, 2022-04-25
• The tutorial takes place every Monday from 14:15 to 15:45 in room 005, Takustr. 9
  • The first tutorial is on Monday, 2022-04-25
  • Note: The first practice sheet will be available on Monday, 2022-04-25 and due to Sunday, 2022-05-01 to be discussed in the tutorial.
• Written exam: Monday, 2022-07-25, 09:59, Takustr. 9, room 049
• Post-exam review (Klausureinsicht): Wednesday, 2022-10-19, 15:59 until at least 16:30, room 49, Takustr. 9

Language

• The course language is German, but the web page, slides, and practice sheets are in English.
• The exam will be formulated in German, but answers may be given in English, too.

Examination modalities

Necessary criteria for obtaining the credit points:

• Completion of at least 80% of the tasks on the practice sheets
• active participation in the tutorial
• passing of the written examination (dictionary may be used)

---

Content

Lecture topics

Foreword: About Inverted Classroom

1. (2022-04-25, this is week 2) The role of empiricism (PDF, TOC+video)
2. (2022-05-02) Study quality: Credibility and relevance (PDF, TOC+video)
3. (2022-05-09) Survey: (Video 2018-05)
   • example: relevance of different topics in Informatics education (article)
   • method: selection of aims; selection of group to be interviewed; design and validation of the questionnaire; execution of the survey; evaluation; interpretation
4. (2022-05-16) Grounded Theory Methodology 1 (PDF, TOC+video)
5. (2022-05-30, due to travel, this is week 7) Grounded Theory Methodology 2 (PDF, TOC+video)
6. (2022-06-13, due to holiday June 6, week 9) Controlled experiment: (Video 2014-05)
   • example 1: flow charts versus pseudo-code (article, criticized prior work)
   • method: control and constancy; problems with reaching constancy; techniques for reaching constancy
   • example 2: use of design pattern documentation (article)
7. (2022-06-13, double package!) Quasi experiment: (Video 2018-06)
   • example 1: comparison of 7 programming languages (article, detailed technical report)
   • method: like controlled experiment, but with incomplete control (mostly: no randomization)
   • example 2: influence of work place conditions on productivity (article)
8. (2022-06-20, week 10) How to lie with statistics: (Video 2018-04)
   • When looking at somebody else's conclusions from data: What is actually meant? What specifically? How can they know it? What is not said?
   • Does the measurement distort the meaning? Is the sample biased?, etc.
   • Material: book on the topic; Study on alternative ink; article with arguments against hypothesis testing: "The earth is round (p < 0.05)".
9. (2022-06-20, triple package!) Data analysis - basic terminology: (Video 2018-06)
   • steps: make data available; validate data; explore data; analyze data (measure, model or compare). Some visualization techniques.
   • example data set (TPC tpmC): raw data (original format), raw data (prepared manually), Import procedure with recoding
10. (2022-06-20, triple package!) Data analysis - techniques: (Video 2018-06, very low-quality audio)
    • Samples and populations; average value; variability; comparison of samples: significance test, confidence interval; bootstrap; relations between variables: plots, linear models, correlations, local models (loess)
    • Article: "A tour through the visualization zoo"
11. (2022-06-27) Case study: (Video 2018-07)
    • example 1: Familiarization with a software team (article)
    • method: characteristics of case studies; what is the 'case'?; use of many data types; triangulation; validity dimensions
    • example 2: An unconventional method for requirements inspections (article)
12. (omitted) Benchmarking: (Video 2018-06)
    • example 1: SPEC CPU2000 (article)
    • Benchmark = measurement + task + comparison; problems (costs, task selection, overfitting); quality characteristics (accessibility, effort, clarity, portability, scalability, relevance) (article)
    • example 2: TREC (article)
13. (2022-07-04) Other methods: (Video 2018-07)
    • The method landscape; simulation; software archeology (studies on the basis of existing data); literature study;
    • example simulation: scaling of P2P file sharing (article)
    • example software archeology: code decline (article)
    • example literature study: a model of the effectiveness of reviews (article)
14. (2022-07-11) Summary and advice: (Video 2018-07)
    • Role of empiricism; quality criteria; generic method; advantages and disadvantages of the methods; practical advice (for data analysis; for conclusion-drawing; for final presentation); outlook

Aims of the tutorials

• Tutorial 1 to 3 (concerning R)
  • To get to know the possibilities of a free, comprehensive and modern statistics software and gain basic skills with it.
  • To get to know a new way of thinking for programming ("programming with data") and practice it.
  • Realize how enlightening a data analysis may be in some cases and how useless in others.
• Tutorial 4 to 13 (project: empirical study)
  • To have gone through the design process of an empirical study oneself and to realize how many aspects must be considered.
  • To experience how many good ideas you may have and how many others possibly are still missing.
  • To realize how important it is to work accurately (because a correction of mistakes is often impossible and usually causes a huge amount of extra work).
  • To have had the gee-whiz experience of analyzing data which nobody else in this world has seen so far.

Practice sheets

(These links will be added continuously as the course proceeds.)

• Preparation for the tutorial: install R
  • version 3.4.4, see r-project.org.

• Practice sheet 1: get to know the data evaluation software "R",
  • R_intro_session.r, for Task 1-3
  • Screencast of that session (Linus Ververs using RStudio, 2022, 42 minutes)
  • ditto (Lutz Prechelt using Emacs-Speaks-Statistics, 2007, 92 minutes, with more explanation)
• Practice sheet 2: practice R, part 1
  • data.zip (contains junit.tsv, junit20.tsv, zile.tsv, jikes.tsv)
• Practice sheet 3: practice R, part 2
• Practice sheet 4: practice R, part 3; Survey: topic
• Practice sheet 5: Survey: method and first version of questionnaire
• Practice sheet 6: Survey: revise questionnaire
• Practice sheet 7: Survey: polish questionnaire
• Practice sheet 8: Survey: pilot test, recruit participants
• Practice sheet 9: Survey: interim report, Evaluating research: Experiments
• Practice sheet 10: Survey: validate data, prepare analysis
• Practice sheet 11: Evaluating research: Quantitative Data Analysis
• Practice sheet 12: Survey: data analysis, final presentation
• Practice sheet 13: Survey: final report
  • LaTeX template

Changes over the years

(minor changes are made every year)

• 2004: Lecture first held.
• 2005: Lecture: only minor changes. Tutorial: broader choice of topics for the surveys.
• 2010: Lecture and tutorial both held in English.
• 2014: Started recording live lecture videos; not all recordings are initially successful. Attempted to add the missing ones each subsequent year.
• 2017: 2nd example with qualitative+quantitative survey added in unit "Surveys"
• 2018: Remaining lecture videos added
• 2022:
  • Course renamed into "Empirical methods in software engineering".
  • Units 1, 2, and 4 replaced by new units 1+2
  • New double unit on GTM added
  • These four also use the new pomalevi video format.
  • Inverted Classroom mode introduced.

Literature

• Set CRAN = https://cran.r-project.org/

• see slides and links in the content list
• R-Dokumentation, especially An Introduction to R and Statistical Computing - Einführung in R
• Bad Science An English language blog written by Dr. Ben Goldacre, which criticises poorly-constructed science. Browsing through it will provide you with many good examples of how not to conduct empirical studies. Contains links to many other similar websites.

---

Comments

Should you have comments or suggestions concerning this page, you may add them here (possibly with date and name):