First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. In this theory paper, we address this common practice in two ways. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability-or quality-of our work to different stakeholders, including funding agencies and the public.

Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies.

In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education.

Qualitative reliability