Arab World English Journal (AWEJ) Special Issue on CALL Number 9. July 2023 Pp.3- 17
DOI: https://dx.doi.org/10.24093/awej/call9.1
ChatGPT’s Capabilities in Spotting and Analyzing Writing Errors Experienced by EFL
Learners
Jeehaan Algaraady
English Department., Faculty of Education, Taiz University, Taiz, Yemen
Mohammad Mahyoob
Languages & Translation Dept. Taibah University, Madina, Saudi Arabia
&
Technical Community College -Taiz, Yemen
Corresponding Author: mqassem@taibahu.edu.sa
Received: 04/02/ 2023 Accepted:07/03/2023 Published:07/24/2023
Abstract:
The recent Large Language Models (LLMs) use advanced algorithms to identify areas where sentence structure and word choice can be improved and to detect grammar, syntax, and spelling mistakes in sentences. This study aimed to investigate the effectiveness of the Chat Generative Pre-trained Transformer (ChatGPT) in detecting English as a foreign language (EFL) learners’ writing errors compared to human instructors. This study examines the ChatGPT as a recent and advanced LLM in analyzing and processing EFL learners’ writing issues. This paper provides valuable insights into the potential benefits and challenges of integrating Artificial Intelligence (AI) into EFL writing education. Our results revealed that ChatGPT successfully identified most surface-level errors but could not detect writing errors related to deep structure and pragmatics. Conversely, human teachers could spot most of these issues. These findings suggest that while ChatGPT can be a valuable tool in identifying surface-level errors, it cannot replace human instructors’ expertise and nuanced understanding in detecting errors related to the more complex aspects of writing. The writing error types (data) are statistically analyzed. The descriptive analysis displays valuable insights into the reliability of the data and its potential implications, where the F-score, which measures the statistical model accuracy, is found to be 1.5. In the meantime, the p-value score, which shows the probability of obtaining results as extreme as the detected data, is calculated to be 0.23. The results suggest that the collected data is statistically significant, and further analysis may yield valuable insights.
Keywords: Artificial Intelligence, ChatGPT, EFL writing, EFL learners, LLMs, error analysis
Cite as: Algaraady, J., & Mahyoob, M.(2023). ChatGPT’s Capabilities in Spotting and Analyzing Writing Errors Experienced by EFL Learners. Arab World English Journal (AWEJ) Special Issue on CALL (9)3-17.
DOI: https://dx.doi.org/10.24093/awej/call9.1
References
Abdalkader, S. M. A. (2022). Using Artificial Intelligence to improve Writing Fluency for The
Preparatory Stage Students in Distinguished Governmental Language Schools. Egyptian
Journal of Educational Sciences, 2(2), 39-70.
Al-Garaady, J., & Mahyoob, M. (2021). Social network communication: Emojis and EFL learners’ writing issues. TESOL International, 16(3.1).
Algaraady, J., & Alrahaili, M. (2022). An Investigation of Parental Perspectives on the
Efficiency of Online Schooling in Primary Schools During the COVID-19
Outbreak. International Journal of Information and Communication Technology Education
(IJICTE), 18(1), 1-18.
Baskara, F. R. (2023). Integrating ChatGPT into EFL writing instruction: Benefits and
challenges. International Journal of Education and Learning, 5(1), 44-55.
Bishop, L. (2023). A computer wrote this paper: What Chatgpt means for education, research,
and writing. Research, and Writing (January 26, 2023).
Bowman, Emma (December 19, 2022). A new AI chatbot might do your homework for you. But it’s
still not an A+ student. NPR. Archived from the original on January 20, 2023. Retrieved December 19, 2022.
Baskara, R. (2023). Integrating ChatGPT into EFL Writing Instruction: Benefits and
Challenges. International Journal of Education and Learning, 5(1).
Chawla, Raveen (December 26, 2022). What is ChatGPT? History, Features, Uses, Benefits, Drawbacks 2023.
Archived from the original on January 7, 2023. Retrieved December 27, 2022.
Chen, T. (2016). Technology-supported peer feedback in ESL/EFL writing classes: A research
synthesis. Computer Assisted Language Learning, 29(2), 365-397.
Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-
based reciprocal peer review system. Computers & Education, 48(3), 409-426.
Corder, S.P. 1981. Error Analysis and Interlanguage. Oxford: Oxford University Press.
Ellis, R. 2002. Second Language Acquisition. Oxford: Oxford University Press
Gayed, J. M., Carlon, M. K. J., Oriola, A. M., & Cross, J. S. (2022). Exploring an AI-based
writing assistant’s impact on English language learners. Computers and Education: Artificial
Intelligence, 3, 100055.
Halaweh, M. (2023). ChatGPT in education: Strategies for responsible implementation.
Contemporary Educational Technology 15(2).
Han, J., Yoo, H., Kim, Y., Myung, J., Kim, M., Lim, H., … & Oh, A. (2023). RECIPE: How to
Integrate ChatGPT into EFL Writing Education. arXiv preprint arXiv:2305.11583.
Holmes, W., Bialik, M., & Fadel, C. (2023). Artificial intelligence in education. Globethics
Publications.
Hong, W. C. H. (2023). The impact of ChatGPT on foreign language teaching and learning:
opportunities in education and research. Journal of Educational Technology and
Innovation, 5(1).
Karim, A., Mohammed A.R., Ismail, SAMM, Shahed, F.H., Rahman, M.M., & Haque, M.H.
- Error Analysis in EFL Writing Classroom. International Journal of English
Linguistics, Vol. 8 (4): 122-138.
Lin, V., Liu, G. Z., & Chen, N. S. (2022). The effects of an augmented-reality ubiquitous writing
application: A comparative pilot project for enhancing EFL writing instruction. Computer
Assisted Language Learning, 35(5-6), 989-1030
Lu, X. (2019). An empirical study on the artificial intelligence writing evaluation system in
China CET. Big data, 7(2), 121-129.
Mahyoob, M. (2021). Online learning effectiveness during the COVID-19 pandemic: A case
study of Saudi universities. International Journal of Information and Communication
Technology Education (IJICTE), 17(4), 1-14.
Park, J. (2019). An AI-based English grammar checker vs. human raters in evaluating EFL
learners’ writing. Multimedia-Assisted Language Learning, 22(1), 112-131.
Park, J. (2019).’ Implications of AI-based grammar checker in EFL learning and testing: Korean
high school students’ writing. The Korea English Language Testing Association, 14(1), 11-39.
Pedro, F., Subosa, M., Rivas, A., & Valverde, P. (2019). Artificial intelligence in education:
Challenges and opportunities for sustainable development. Paris: UNESCO.
Perkins, M. (2023). Academic Integrity considerations of AI Large Language Models in the post-
pandemic era: ChatGPT and beyond. Journal of University Teaching & Learning
Practice, 20(2), 07.
Shi, H & Aryadoust, V. (2023) A systematic review of automated writing evaluation
systems.” Education and Information Technologies 28. (1): 771-795.
Stevenson, M., & Phakiti, A. (2019). Automated feedback and second language
writing. Feedback in second language writing: Contexts and issues, 125-142.
Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing
classrooms. Assessing Writing, 57, 100752.
Tate, T. P., Doroudi, S., Ritchie, D., Xu, Y., & uci, m. w. (2023, January 10). Educational
Research and AI-Generated Writing: Confronting the Coming Tsunami.
https://doi.org/10.35542/osf.io/4mec
Wang, Z. (2022). Computer-assisted EFL writing and evaluations based on artificial intelligence:
a case from a college reading and writing course. Library Hi Tech, 40(1), 80-97.
Wu, H., Wang, W., Wan, Y., Jiao, W., & Lyu, M. (2023). Chatgpt or grammarly? evaluating
chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648.
Zhai, N., & Ma, X. (2023). The Effectiveness of Automated Writing Evaluation on Writing
Quality: A Meta-Analysis. Journal of Educational Computing Research, 61(4), 875–
Yan, D. (2023). How ChatGPT’s automatic text generation impact on learners in a L2 writing
practicum: an exploratory investigation. ⟨hal-04037687⟩
Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory
investigation. Education and Information Technologies, 1-25.