Awarded Best Paper, one of two such awards out of 95 submissions and 26 accepted papers.

Abstract

To teach students software engineering skills, instructors must be able to explain what quality looks like and be able to assess it. Operationally, this means that we need comprehensive rubrics for assessing artifacts such as commits, issues, pull requests and code reviews. For software engineering education research, such rubrics are also needed for assessing the effectiveness of various interventions. Ideally, these rubrics would be rooted in some ground truth, rather than reflecting the subjective opinions of individual instructors; this is particularly important if rubrics are used in empirical studies. In this paper, we seek to establish a “ground truth” for the assessment of commits by using the Delphi process, an empirical method for establishing expert consensus around a complex question. We assembled a diverse panel of 16 experts in software engineering with voices from both industry and academia. By applying the Delphi process through four rounds of questions to our panel, we derived a set of rubric items for commits that represent the consensus view of the expert panel. Our contribution is both the rubric items themselves and a roadmap by which future researchers can arrive at similar rubrics for other software engineering artifacts.