AI-Powered Automated Speaking Scoring and Human Evaluation: A Comparative Study of EFL Learners’ Oral Proficiency

References:

Alam, S., Amir, Saudi, T. T., Ahmad, F., Kralik, R., Rasheed, T. (2026a). Impact of Meta AI           generated corrective feedback in the writing classroom: effects on L2 writing linguistic        errors. World Journal of English Language, 16(3), 119-130.            https://doi.org/10.5430/wjel.v16n3p119.

Alam, S., Ahmad, F., Mohamed, A. O., Dawood, J. A., Obloberdiyevna, D. S., & Rasheed, T.     (2026b). Artificial intelligence in Saudi English language classrooms: A narrative      review. World Journal of English Language, 16(5),            254. https://doi.org/10.5430/wjel.v16n5p254

Alam, S., Hameed, A., Balogh, Z., & Haque, S. A. (2025). Digitalisation of education:            challenges and issues in virtual pedagogical competence. Journal of Education            Culture and Society, 16(1), 165-177. https://doi.org/10.15503/jecs2025.2.165.177

Bamdev, P., Grover, M. S., Singla, Y. K., Vafaee, P., Hama, M., & Shah, R. R. (2023).            Automated speech scoring system under the lens: evaluating and interpreting the       linguistic cues for language proficiency. International Journal of Artificial Intelligence     in Education, 33(1), 119–154. https://doi.org/10.1007/s40593-022-00291-5

Benaissa, M., Attou, Y., Seddik, M., & Nfissi, A. (2025). Can AI-based teacher-bots (T-bots)           enhance students’ speaking skills? A case study of “My AI” on Snapchat in Moroccan            higher education. Arab World English Journal (AWEJ), Special Issue on CALL(11),    144–159. https://doi.org/10.24093/awej/call11.9

Brown, H. D., Abeywickrama, P. (2021). Language assessment: principles and classroom        practices. Pearson.

Butarbutar, R., Sujarwo, Gunu Pure, E. A., Sulviana, & Rizal, A. (2025). Unlocking the impact       of AI-assisted online interactive learning environment in English language learning.      The Journal of Asia TEFL, 22(4), 646–662.            https://doi.org/10.18823/asiatefl.2025.22.4.2.646

Clapham, C. (2000). Measuring second language performance.: Tim McNamara. TESOL Quarterly, 34(2), 376-377. https://doi.org/10.2307/3587967

Crossley, S. & McNamara, D. (2013). Applications of text analysis tools for spoken            response grading. Language Learning & Technology, 17(2), 171–192.            https://doi.org/10.64152/10125/44329

Davis, L., & Papageorgiou, S. (2021). Complementary strengths? Evaluation of a hybrid human-machine scoring approach for a test of oral academic English. Assessment in           Education: Principles, Policy & Practice, 28(4), 437–455.     https://doi.org/10.1080/0969594X.2021.1979466

Eckes, T. (2015). Introduction to Many-Facet Rasch Measurement. Analyzing and Evaluating      Rater-Mediated Assessments. Berlin, Germany: Peter Lang Verlag. Retrieved on Aug        20, 2026. https://doi.org/10.3726/978-3-653-04844-5

Fang, Y. H., Lo, T. H., Sung, Y. T., & Chen, B. (2025, December). Beyond Modality            Limitations: A Unified MLLM Approach to Automated Speaking Assessment with     Effective Curriculum Learning. In 2025 IEEE Automatic Speech Recognition and      Understanding Workshop (ASRU) (pp. 1-7). IEEE. (Scopus Indexed)

Fulcher, G. (2015). Re-examining language testing: A philosophical and social inquiry.           Routledge. Scopus Indexed

Giavarina, D. (2015). Understanding Bland-Altman analysis. Biochemia medica, 25(2), 141-    151. https://doi.org/10.11613/BM.2015.015

Grab, M. O. (2026). Embedding design thinking and artificial intelligence in EFL classrooms:     Effects on secondary learners’ oral proficiency. Social Sciences & Humanities Open,       14, 103465. https://doi.org/10.1016/j.ssaho.2026.103465

Hu, H., & Hashim, H. (2026). What shapes automated ratings in computer-based English            speaking tests? Perspectives from analytic complexity, accuracy, fluency, and    pronunciation indices. VIAL-Vigo International Journal of Applied Linguistics.     23(1), 51-82. https://doi.org/10.35869/vial.v0i23.6467

Hu, H., Said, N. E. M., & Hashim, H. (2025). Human ratings and complexity, accuracy, and  fluency (CAF) indices: A correlational study of a standardised monologic English     speaking test in China. SAGE Open, 15(2), 1-13.            https://doi.org/10.1177/21582440251343944

Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of   L2 pronunciation: revisiting research conventions. Language Assessment      Quarterly, 10(2), 135–159. https://doi.org/10.1080/15434303.2013.769545

Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of            Educational Measurement, 50(1), 1-73. https://doi.org/10.1111/jedm.12000

Karatay, Y. and Xu, J. (2025), Exploring the potential of conversational AI for assessing second         language oral proficiency. TESOL Quarterly, 59(1), S220-S250.            https://doi.org/10.1002/tesq.70003

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., &            Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large    language models for education. Learning and Individual Differences, 103, 102274.           https://doi.org/10.1016/j.lindif.2023.102274

Kobylarek, A., & Birova, J. (2026). AI as the philosophy of a matrix society. educational     implications. Journal of Education Culture and Society, 17(1), 9-18.            https://doi.org/10.15503/jecs2026.1.9.18

Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and            learning. Relc Journal, 54(2), 537-550. https://doi.org/10.1177/00336882231162868

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation      coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163.      https://doi.org/10.1016/j.jcm.2016.02.012

Lai, Z. C.-C. (2026). AI in EFL blended learning: Impact on speaking performance and learning    resilience. Asian-Pacific Journal of Second and Foreign Language Education, 11, 11.       https://doi.org/10.1186/s40862-025-00367-4

Li, H. (2024). Design and development of an oral English evaluation system based on speech recognition technology and a scoring algorithm. Journal of Electrical Systems, 20(6),  75-84.  https://doi.org/10.52783/jes.2616

Li, J., Huang, J., & Sheeran, T. J. (2025). ChatGPT-4o as an AI peer assessor in EFL speaking          classrooms: Examining scoring reliability and feedback effectiveness. SAGE Open,    15(3), 1-15. https://doi.org/10.1177/21582440251369938

Li, J., Yan, B., Lo, T., Wang, Y., Hsu, Y., & Chen, K. (2024). Automated speaking assessment      of conversation tests with novel graph-based modeling on spoken response coherence.    IEEE Spoken Language Technology Workshop (SLT), Macao, pp. 825-832.    https://doi.org/10.1109/SLT61566.2024.10832136.

Linacre, J. M. (1989). Rasch Models from Objectivity: A Generalization. Retrieved from    https://eric.ed.gov/?id=ED308211

Matsuura, R., Suzuki, S. (2023). Prompt-Independent Automated Scoring of L2 Oral Fluency           by Capturing Prompt Effects. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N.,   Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023.     Lecture Notes in Computer Science, vol 13916. Springer, Cham.            https://doi.org/10.1007/978-3-031-36272-9_62

Mendonça, P. C., Quintal, F., & Mendonça, F. (2025). Evaluating LLMs for automated scoring in formative assessments. Applied Sciences, 15(5), 2787-2805.            https://doi.org/10.3390/app15052787

Messick, S. (1989). Validity. In R. linn (org.), Educational Measurement.(13-103). New York,          NY: American Council on Education and Macmillan Publishing Company.

Metruk, R. (2018). Comparing holistic and analytic ways of scoring in the assessment of         speaking skills. Journal of Teaching English for Specific and Academic Purposes, 6(1),           179–189. https://doi.org/10.22190/JTESAP1801179M

Mohammadkarimi, E., Omar, J. A., & Rashid, A. S. (2025). Advancements and Challenges in  Automated Evaluation of Spoken Language Proficiency Using AI. In N. Jomaa         (Ed.), Using AI Tools in Text Analysis, Simplification, Classification, and Synthesis (pp.       225-256). IGI Global Scientific Publishing. https://doi.org/10.4018/979-8-3693-9511      0.ch008

Ndacyayisenga, J. C. Y., Kidega, C., & Can, L. A. (2026). The impact of generative artificial          intelligence tools on assessment equity for non-native English-speaking medical        students: A systematic review. BMC Medical Education, 26, 989.     https://doi.org/10.1186/s12909-026-09303-7

Nikolovski, V., Trajanov, D., & Chorbev, I. (2025). Advancing AI in higher education: a          comparative study of large language model-based agents for exam question            generation, improvement, and evaluation. Algorithms, 18(3), 1-34.            https://doi.org/10.3390/a18030144

Obera, J. A., Ojochegbe, A. T., Obera, J. A., & Ojochegbe, A. T. (2025). Automated scoring systems and teacher assessments in English language education. Journal of World  Science, 4(10), 1352–1358. https://doi.org/10.58344/jws.v4i10.1517

Ockey, G. J., & Chukharev-Hudilainen, E. (2021). Human versus computer partner in the       paired oral discussion test. Applied Linguistics, 42(5), 924–944.            https://doi.org/10.1093/applin/amaa067

Ogawa, C. (2022). CAF indices and human ratings of oral performances in an opinion-based monologue task. Language Testing in Asia, 12(1), 1-18.            https://doi.org/10.1186/s40468-022   00154-9

Polasa, P., Laoaree, S., Thanadunpremdet, T., Rodjananant, N., & Kritsuthikul, N. (2025). “Chai   Calibrated Hybrid Assessment for IELTS Speaking with Human-Referenced      Validation,” 2025 20th International Joint Symposium on Artificial Intelligence and          Natural Language Processing (iSAI-NLP), Phuket, Thailand, 2025, pp. 1-6, doi:      10.1109/iSAI-NLP66160.2025.11320542.

Scaria, N., Kennedy, S. J. J., Latinovich, T., & Subramani, D. N. (2024). Evalyaks: Instruction      tuning datasets and LoRA fine-tuned models for automated scoring of CEFR B2         speaking assessment transcripts. Computers and Education: Artificial Intelligence,    10(2026), 1-20. https://doi.org/10.1016/j.caeai.2025.100539

Shikun, S., Grigoryan, G., Huichun, N., & Harutyunyan, H. (2024). AI chatbots: Developing     English language proficiency in EFL classroom. Arab World English Journal (AWEJ),         Special Issue on ChatGPT, 292–305. https://doi.org/10.24093/awej/ChatGPT.20

Wang, X., Evanini, K., Qian, Y., & Mulholland, M. (2021). Automated scoring of spontaneous    speech from young learners of English using transformers. 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China, 2021, pp. 705-712.             https://doi.org/10.1109/SLT48900.2021.9383553

Wang, Y., & Min, S. (2026). The human–machine correlation in automated speech evaluation:      A three-level meta-analysis. Language Testing, 43(2), 165-196.            https://doi.org/10.1177/02655322251387816

Wang, Z., & Sun, Y. (2018). Comparison of human rater and automated scoring of test takers’ speaking ability and classification using item response theory. Psychological Test and           Assessment Modeling, 60(1), 81-100.

Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of         automated scoring. Educational measurement: issues and practice, 31(1), 2-13.

Wind, S. A., & Peterson, M. E. (2018). A systematic review of methods for evaluating rating  quality in language assessment. Language Testing, 35(2), 161-192.            https://doi.org/10.1177/0265532216686999

Xi, X. (2010). Automated scoring and feedback systems: Where are we and where are we       heading?. Language Testing, 27(3), 291-300.            https://doi.org/10.1177/0265532210364643

Xi, X., Higgins, D., Zechner, K., & Williamson, D. M. (2008). Automated scoring of            spontaneous speech using SpeechRaterSM v1.0. ETS Research Report Series, 2008(2),          1-113.  https://doi.org/10.1002/J.2333-8504.2008.TB02148.X

Xu, J., Jones, E., Laxton, V., & Galaczi, E. (2021). Assessing L2 English speaking using   automated scoring technology: examining automarker reliability. Assessment in         Education: Principles, Policy & Practice, 28(4), 411–436.            https://doi.org/10.1080/0969594X.2021.1979467

Yavuz, F., Çelik, Ö., & Yavaş Çelik, G. (2025). Utilizing large language models for EFL essay       grading: An examination of reliability and validity in rubric‐based assessments. British   Journal of Educational Technology, 56(1), 150-166. https://doi.org/10.1111/bjet.13494

Zechner, K., & Bejar, I. I. (2006). Towards automatic scoring of non-native spontaneous    speech. Human Language Technology Conference of the North American Chapter of  the Association of Computational Linguistics, Proceedings, June 4-9, 2006, New York,     New York, USA. https://doi.org/10.3115/1220835.1220863

Zechner, K., Evanini, K., Yoon, S., Davis, L., Wang, X., Chen, L., Lee, C. M., & Leong, C. W.   (2014). Automated scoring of speaking items in an assessment for teachers of English     as a foreign language. Proceedings of the Ninth Workshop on Innovative Use of NLP           for Building Educational Applications. Baltimore, Maryland USA.           https://doi.org/10.3115/V1/W14-1816

Zhong, W., Zhou, Z., Wang, L., & Yang, Y. (2026). Developing and validating multi-scenario          EFL willingness to communicate scale across classroom, extracurricular, and GenAI     mediated scenarios. BMC Psychology, 14, 819. https://doi.org/10.1186/s40359-026   04607-1

Zhou, T., & Yu, H. (2025). Research on deep learning-based English speaking automatic        scoring system. CISAI 2025: 2025 8th International Conference on Computer        Information Science and Artificial Intelligence, China.             https://doi.org/10.1145/3773365.3773381

Zou, B., Du, Y., Wang, Z., Chen, J., & Zhang, W. (2023). An investigation into artificial          intelligence speech evaluation programs with automatic feedback for developing EFL          learners’ speaking skills. SAGE Open. 13(3), 1-9.            https://doi.org/10.1177/21582440231193818