Development and Validation of a Scale for Assessing the Effectiveness of Generative AI-Enhanced English Writing Learning Among Chinese University Students

Abstract

This study, grounded in the Technology Acceptance Model (TAM) and student engagement theory, aims to develop a psychometric instrument for assessing the effectiveness of generative AI-enhanced English writing learning among Chinese university students. Drawing upon established scales from domestic and international literature, the initial scale was adapted to the specific context of generative AI-assisted English writing, comprising three dimensions—AI Use Attitude, Learning Engagement, and Learning Outcomes—with 42 items. Following expert review, the scale was administered to university students across 28 provinces in China, yielding 282 valid responses. Data were analyzed using SPSS Statistics 27, employing item analysis, exploratory factor analysis (EFA), and reliability and validity testing. Item analysis, based on critical ratio (CR), corrected item-total correlation (CITC), communality, and factor loading, resulted in the deletion of six items (Q8, Q15, Q27, Q35, Q39, Q42), retaining 36 items in the final scale. Exploratory factor analysis supported a three-dimensional structure: “AI Use Attitude—Learning Engagement—Learning Outcomes.” Reliability analysis demonstrated Cronbach's α coefficients exceeding 0.87 for all dimensions and the overall scale, with a split-half reliability of 0.947. Validity testing confirmed satisfactory content validity and structural validity. The findings indicate that the scale possesses robust psychometric properties and practical feasibility, providing a valid instrument for evaluating the effectiveness of generative AI-enhanced English writing instruction.