The value of Generative AI for Qualitative Research: A Pilot Study

Abstract

This mixed methods approach study investigates the potential of introducing Generative AI (ChatGPT 4 and Bard) as part of a deductive qualitative research design that requires coding, focusing on possible gains in cost-effectiveness, coding throughput time and inter-coder reliability (Cohen’s Kappa). This study involved semi-structured interviews with five domain experts and analyzed a dataset of 122 respondents that required categorization into six pre-defined categories. The results from using Generative AI coders were compared with those from a previous study where human coders carried out the same task. In this comparison, we evaluated the performance of AI-based coders against two groups of human coders, comprising three experts and three non-experts. Our findings support the replacement of human coders with Generative AI ones, specifically ChatGPT for deductive qualitative research methods of limited scope. The experimental group, consisting of three independent Generative AI coders, outperformed both control groups in coding effort, with a fourfold (4x) efficiency and throughput time (15x) advantage. The latter could be explained by leveraging parallel processing. Concerning expert vs. non-expert coders, minimal evidence suggests a preference for experts. Although experts code slightly faster (17%), their inter-coder reliability showed no substantial advantage. A hybrid approach, combing ChatGPT and domain experts shows the most promise. This approach reduces costs, shortens project timelines, and enhances inter-coder reliability, as indicated by higher Cohen's Kappa values. In conclusion, Generative AI, exemplified by ChatGPT, offers a viable alternative to human coders, in combination with human research involvement, delivering cost savings and faster research completion without sacrificing notably reliability. These insights, while limited in scope, show potential for further studies with lager datasets, more inductive qualitative research designs and other research domains.

Received: 30 March 2024 | Revised: 17 May 2024 | Accepted: 18 June 2024 

Conflicts of Interest

The author declares that he has no conflicts of interest to this work. 

Data Availability Statement

The data that support the findings of this study are not publicly available due to privacy concerns. However, anonymous data are available on reasonable request. Requests should be made to Frédéric Pattyn(Frederic.Pattyn@ugent.be) and should include a brief description of the intended use of the data.