Motivation

Reproducibility of research in computer science and in the field of networking in particular is a well-recognized problem. For several reasons, including the sensitive and/or proprietary nature of some Internet measurements, the networking research community discounts the importance of reproducibility of results, instead tending to accept papers that appear plausible. Studies have shown that a fraction of published papers release artefacts (such as code and datasets) that are needed to reproduce results. To provide incentives to authors, conferences attempt to bestow best dataset awards and actively solicit submissions that reproduce results. Community archives (such as DatCat and CRAWDAD) exist that provide an index of existing measurement data and invite the community to reproduce existing research. SIGCOMM Computer Communication Review allows authors to upload artefacts during the paper submission page to allow reviewers to check for reproducibility, and relaxes page limits for reproducible papers. Association for Computing Machinery (ACM) has lately also taken an initiative and introduced a new policy on result and artefact review and badging. The policy sets a terminology to use to assess results and artefacts. ACM has also initiated a new task force on data, software and reproducibility in publication to understand how ACM can effectively promote reproducibility within the computing research community. Despite these continued efforts, reproducibility of research in computer science and in the field of networking in particular appears to exist as an ongoing problem since papers that reproduce existing research rarely get published in practise.

In this Seminar, we aim to discuss challenges to improving reproducibility of scientific Internet research, and hope to develop a set of recommendations that we as a community can undertake to initiate a cultural change toward reproducibility of our work.