Discovering Levels of Cognitive Demand in Tests Developed by Iranian High School English Teachers across Grades

Abstract

Despite the significance of the cognitive load of educational procedures, its status quo in teacher-made tests has remained largely uncharted. This study investigated the interplay between cognitive demand levels (CDL) of test items and students' academic levels. Using the content analysis method, English language tests developed by Iranian high school teachers across three grades (10 through 12) were collected and a scheme was developed to code the CDL of the tests. The coding scheme was developed based on the Revised Bloom’s Taxonomy (RBT), Webb's Depth of Knowledge (DoK), and Hess's cognitive Rigor Matrix (CRM). Three trained coders coded the tests using the piloted and validated scheme. The results showed that the low CDLs of “remembering,” “understanding,” and “applying” constituted a large proportion (89.49%), while “Analyzing,” “Evaluating,” and “Creating,” as the high CDLs comprised a small number (10.51%) of the test items. Grade 12 and grade 11 tests contained the most frequent high and low CDLs, respectively. In all grades, “understanding” was the most frequent level and “creating” was the least frequent level. The trend of Remembering and Creating was normal, Understanding and Analyzing was descending and Applying and Evaluating showed variation across the grades. The study concludes that the tests predominantly required low CDLs and were not developed based on a measure of cognitive demand balancing and/or regulating low and high CDLs across the grades. It recommends that test developers use a cognitive demand guide specifying the underlying nature of tasks required in the tests, the complexity of the expressions and information presented in the tests, and the choice of test item stem format and response format to regulate the cognitive demand of tests for different school grades.

Received: 12 August 2024 | Revised: 29 November 2024 | Accepted: 3 March 2025

Conflicts of Interest

The authors declare that they have no conflicts of interest to this work.

Data Availability Statement

The data that support this work are available upon reasonable request to the corresponding author.

Author Contribution Statement

Ali Ghezalbash: Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Project administration. Iman Alizadeh: Conceptualization, Methodology, Investigation, Writing - review & editing, Supervision. Homa Divsar: Methodology, Validation, Investigation, Resources, Data curation, Project administration.