Exploring Methodological Challenges in Educational Research: A Case Reflection on Response Surface Analysis

Abstract

The growing availability of open-access analysis scripts has facilitated the use of advanced statistical methods such as response surface analysis (RSA). However, accessibility alone does not ensure appropriate application. In this commentary, we consider a published study as one illustrative case study, showing how the use of shared scripts—even from expert sources—may lead to fragile conclusions if methodological assumptions are overlooked. Through a detailed reconstruction of the modeling process, we identify mismatches between analytic choices and theoretical aims, including issues in model specification, data treatment, and interpretation. A small simulation further illustrates how routine preprocessing decisions, such as handling missing data or centering predictors, can meaningfully alter the fitted surface, underscoring the practical consequences of these choices. We offer practical suggestions to help researchers identify common pitfalls in the use of RSA. We also place these observations in the broader context of open science, highlighting how transparency in analysis must be accompanied by careful scrutiny of modeling assumptions. This reflection aims to support more enlightened analytic practices and encourage cautious engagement with complex statistical techniques. In doing so, we argue that the benefits of open-access tools can be fully realized if paired with clear methodological awareness and sound theoretical reasoning.

Received: 15 October 2025 | Revised: 28 November 2025 | Accepted: 5 February 2026

Conflicts of Interest

The authors declare that they have no conflicts of interest to this work.

Data Availability Statement

The data that support the findings of this study are openly available in OSF at https://osf.io/fcqr2/overview?view_only=82fe0a78a36e4993b03a676c5e14638b.

Author Contribution Statement

Kim Sorjonen: Conceptualization, Methodology, Validation, Formal Analysis, Writing – original draft, Writing – review and editing; Rebeka Weidmann: Conceptualization, Methodology, Validation, Writing – review and editing.