Arab World English Journal (AWEJ) Volume 14. Number 4 December 2023                   Pp.213-232
DOI: https://dx.doi.org/10.24093/awej/vol14no4.13

Full Paper PDF

Topic Familiarity Effects on Performance in Speaking Assessment Tasks

Nurul Iman Ahmad Bukhari
Faculty of Language Studies and Human Development, Universiti Malaysia Kelantan, Kelantan, Malaysia
Faculty of Educational Studies, Universiti Putra Malaysia, Selangor, Malaysia

Lilliati Ismail
Faculty of Educational Studies, Universiti Putra Malaysia, Selangor, Malaysia
Corresponding Author: lilliati@upm.edu.my

 Noor Lide Abu Kassim
Kuliyyah of Education, International Islamic University Malaysia, Kuala Lumpur, Malaysia
Corresponding Author: noorlide@iium.edu.my

Abu Bakar Razali
Faculty of Educational Studies, Universiti Putra Malaysia, Selangor, Malaysia

 Nooreen Noordin
Faculty of Educational Studies, Universiti Putra Malaysia, Selangor, Malaysia

 Muhamad Firdaus Mohd Noh
Sekolah Rendah Agama Bersepadu Segamat, Johor, Malaysia

 

Received:07/11/2023                                Accepted: 11/28/2023                             Published:  12/15/2023

 

Abstract:
Speaking assessment is believed to be difficult in its expansion and execution.   Thus, it is a challenge for teachers in preparing students for speaking tests. This study’s purpose was to identify whether topic familiarity affects speaking performance among students who were preparing to sit for the high-stakes Malaysian University English Test speaking test.  The study aimed to investigate the validity and reliability of the speaking assessment measures and, subsequently, to examine the speaking tests’ item difficulty measures differences according to topic familiarity level.  Data were collected from 40 non-native speakers of English among Malaysian Form Six pre-university students who were preparing for their MUET test. The researcher conducted two practice speaking tests, which used retired papers of the MUET CEFR-aligned speaking tests, to the 40 participants who were grouped into 10 speaking test groups. The practice speaking tests were video and audio-recorded. Topic familiarity was measured using self-report questionnaires.  In the second phase, each of the seven appointed raters scored all 40 students’ responses in two speaking tests consisting of two speaking tasks assessed across four criteria: task fulfillment, language, communicative ability, and group discussion.  Many-facet Rasch measurement results revealed significant differences translating to significant influence of topic familiarity on speaking performance. The present study’s results not only confirmed the significance of topic familiarity in the preparation for speaking assessments, but also highlighted the need for formal teaching on potential topic themes that are commonly encountered in such assessments, particularly those that are at the post-secondary level.  These findings imply importance in designing speaking tests taking into account test-taker topic familiarity.
Keywords: language testing, many-facet Rasch measurement, oral performance, speaking assessment, topic familiarity

Cite as:  Bukhari, N. I.A., Ismail, L., Abu Kassim, N.L., Abu Bakar Razali, A., Noordin, N., & Noh, M. F. M. (2023). Topic Familiarity Effects on Performance in Speaking Assessment Tasks.  Arab World English Journal, 14 (4) 213-232.
DOI: https://dx.doi.org/10.24093/awej/vol14no4.13

References

Abu Bakar, N. I., Noordin, N., & Razali, A. B. (2019). Improving Oral Communicative Competence in English Using Project-Based Learning Activities. English Language Teaching, 12(4), 73–84. https://doi.org/10.5539/elt.v12n4p73

Abu Kassim, N. L., & Zubairi, A. M. (2006). Interaction Between Test-taker Characteristics, Task Facets and L2 Oral Proficiency Test Performance. Educational Awakening: Journal of the Educational Sciences, 3(2), 139–159. http://irep.iium.edu.my/29551/1/29551_interaction_between_test_taker_characteristics.pdf

Afaf Ayed Alrowaithy. (2021). The Effect of Topic Familiarity on Improving EFL Saudi Female Students` Reading Comprehension. Journal of Educational and Psychological Sciences, 5(7), 118–135. https://doi.org/10.26389/ajsrp.b020620

Ahmad Tarmizi, S. A., et al. (2022). A Socio-Cognitive Perspective on the Factors Affecting Malaysian Business Students’ Learning when Spoken in English in a Second-Language Classroom. International Journal of Learning, Teaching and Educational Research, 21(1), 67–91. https://doi.org/10.26803/ijlter.21.1.5

Ameri-Golestan, A. (2016). Effects of Teaching Strategies and Topic Familiarity on Persian Speaking IELTS Candidates’ Speaking Scores. Modern Journal of Language Teaching Methods, 6(3), 44–50. https://www.webofscience.com/wos/woscc/full-record/WOS:000379367100004

Anselmi, P., Colledani, D., & Robusto, E. (2019). A Comparison of Classical and Modern Measures of Internal Consistency.
Frontiers in Psychology, 10(December), 1–12. https://doi.org/10.3389/fpsyg.2019.02714

Aryadoust, V., Ng, L. Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6–40. https://doi.org/10.1177/0265532220927487

Bachman, L. F. (1999). Appendix: Language testing – SLA research interfaces. In L. F. Bachman & A. D. Cohen (Eds.), Interfaces between Second Language Acquisition and Language Testing Research (pp. 177–195). Cambridge University Press. https://doi.org/10.1017/CBO9781139524711.010

Badrasawi, K. J. I., Kassim, N. L. A., Zubairi, A. M., Johar, E. M., & Sidik, S. S. (2021). English Language Speaking Anxiety, Self-Confidence and Perceived Ability among Science and Technology Undergraduate Students: A Rasch Analysis. Pertanika Journal of Social Sciences and Humanities, 29(3), 309–334. https://doi.org/10.47836/pjssh.29.s3.16

Baharudin, H., Maskor, Z. M., & Matore, M. E. E. M. (2022). The raters’ differences in Arabic writing rubrics through the Many-Facet Rasch measurement model. Frontiers in Psychology, 13, 7760. https://doi.org/10.3389/FPSYG.2022.988272/BIBTEX

Banerjee, H. L. (2019). Investigating the Construct of Topical Knowledge in Second Language Assessment: A Scenario-Based Assessment Approach. Language Assessment Quarterly, 16(2), 133–160. https://doi.org/10.1080/15434303.2019.1628237

Bijani, H. (2018). Effectiveness of a Face-to-Face Training Program on Oral Performance Assessment : The Analysis of Tasks Using the Multifaceted Rasch Analysis. 5(4), 27–53. https://doi.org/10.30479/jmrels.2019.10667.1335

Bijani, H. (2019). Evaluating the effectiveness of the training program on direct and semi-direct oral proficiency assessment: A case of multifaceted Rasch analysis. Cogent Education, 6(1), 1670592. https://doi.org/10.1080/2331186X.2019.1670592

Bijani, H., & Khabiri, M. (2017). Investigating the effect of training on raters’ bias toward test takers in oral proficiency assessment: A FACETS analysis. Journal of Asia TEFL, 14(4), 687–702. https://doi.org/10.18823/asiatefl.2017.14.4.7.687

Bond, T. G., & Fox, C. (2015). Applying the Rasch Model; Fundamental Measurement in the Human Sciences |
Request PDF (3rd ed.). Routledge. https://www.researchgate.net/publication/312296223_Applying_the_Rasch_Model_Fundamental_Measurement
_in_the_Human_Sciences

Bonk, W. J., & Ockey, G. J. (2003). A many-facet Rasch analysis of the second language group oral discussion task.
Language Testing, 20(1), 89–110. https://doi.org/10.1191/0265532203lt245oa

Brown, A. (2003). Interviewer variation and the co-construction of speaking proficiency. Language Testing, 20(1), 1–25. https://doi.org/10.1191/0265532203lt242oa

Bui, G., & Huang, Z. (2016). L2 fluency as influenced by content familiarity and planning : Performance , measurement ,
and pedagogy. Language Teaching Research, 22(1), 94–114. https://doi.org/https://doi.org/10.1177/1362168816656650

Chonghui, L. (2019). Boosting English Standards. The Star Online. Available at https://www.thestar.com.my/news/education/2019/06/09/boosting-english-standards

Council of Europe. (2001). the Common European Framework of Reference for Languages : Learning, Teaching, Assessment.
Council of Europe. https://doi.org/10.1017/S0267190514000221

Fischer, J. (2020). The underlying action-oriented and task-based approach of the CEFR and its implementation in language testing and assessment at university. Language Learning in Higher Education, 10(2), 301–316. https://doi.org/10.1515/cercles-2020-2021

Fu, J. S., Yang, S. H., & Yeh, H. C. (2021). Exploring the impacts of digital storytelling on English as a foreign language learners’ speaking competence. Journal of Research on Technology in Education, 0(0), 1–16. https://doi.org/10.1080/15391523.2021.1911008

Geranpayeh, A., & Ahmad Zufrie, A. R. (2018). The Alignment Of Malaysian University English Test To The CEFR. 6th British Council New Directions in English Language Assessment Conference: Standards in Learning Systems, October, 21. Retreived from https://www.britishcouncil.my/sites/default/files/nd_booklet_16_oct_v1.pdf

Han, L., & Md Yusof, M. A. (2019). Students’ Self-reflections of Own Participation in English Language Oral Class. LSP International Journal, 6(2), 79–92. https://doi.org/10.11113/lspi.v6n2.92

Hidri, S. (2018). Assessing spoken language ability: A many-facet Rasch analysis. Second Language Learning and Teaching, 9783319628837, 23–48. https://doi.org/10.1007/978-3-319-62884-4_2

Huang, H. T. D., Hung, S. T. A., & Hong, H. T. V. (2016). Test-Taker Characteristics and Integrated Speaking Test Performance: A Path-Analytic Study. Language Assessment Quarterly, 13(4), 283–301. https://doi.org/10.1080/15434303.2016.1236111

Huang, H. T. D., Hung, S. T. A., & Plakans, L. (2018). Topical knowledge in L2 speaking assessment: Comparing independent and integrated speaking test tasks. Language Testing, 35(1), 27–49. https://doi.org/10.1177/0265532216677106

Idris, M. B., & Abdul Raof, A. H. Bin. (2019). Learner-Driven Oral Assessment Criteria for English Presentation
. Journal of Nusantara Studies (JONUS), 4(1), 365. https://doi.org/10.24200/jonus.vol4iss1pp365-383

Idris, M., & Zakaria, M. H. (2016). Gauging esl learners’ cefr ratings on oral proficiency in rater training. Man in India,
96(6), 1675–1682. Available at https://scholar.googleusercontent.com/scholar?q=cache:Rw6rZgYUSYUJ:scholar.google.com/&hl=id&as_sdt=0,5&scioq=GAUGING+ESL+LEARNERS’+
CEFR+RATINGS+ON+ORAL+PROFICIENCY+IN+RATER+TRAINING+Mardiana+Idris+and+Mohamad+Hassan+Zakaria

Ismail, L., & Abd. Samad, A. (2014). Using Tasks and Repair Practices in the Malaysian L2 Classroom. The Asia-Pacific Education Researcher, 23(3), 499–510. https://doi.org/10.1007/s40299-013-0124-7

Jahedi, M., & Ismail, L. (2020). Factors affecting ESL students’ willingness to communicate in english classroom
discussions and their use of linguistic strategies. Universal Journal of Educational Research, 8(8), 3360–3370. https://doi.org/10.13189/ujer.2020.080808

Khabbazbashi, N. (2017). Topic and background knowledge effects on performance in speaking assessment. Language
Testing, 34(1), 23–48. https://doi.org/10.1177/0265532215595666

Lai Kuen, G., Rafik-Galea, S., & Swee Heng, C. (2017). Effect of Oral Communication Strategies Training on the Development of Malaysian English as a Second Language Learners’ Strategic Competence. International Journal of Education and Literacy Studies, 5(4), 57. https://doi.org/10.7575/aiac.ijels.v.5n.4p.57

Lateh, N. H. M., Shamsudin, S., & Said, S. M. (2015). Influence of Malaysian University English Test Training on the Speaking Performance of Pre-University Students. Advanced Science Letters, 21(7), 2463–2465. https://doi.org/10.1166/asl.2015.6311

Linacre, J. M. (2023). Facets computer program for many facet Rasch measurement (Version 3.87.0). https://www.winsteps.com

Lumley, T., & O’Sullivan, B. (2005). The effect of test-taker gender, audience and topic on task performance in tape-mediated assessment of speaking. Language Testing, 22(4), 415–437. https://doi.org/10.1191/0265532205lt303oa

Malaysian Examinations Council. (2019). Malaysian University English Test (MUET). In Batu Caves: Percetakan Warni. Retreived from https://www.mpm.edu.my/images/dokumen/calon-peperiksaan/muet/regulation/Test_Specification_Regulation.pdf

Mohd Noh, M. F. Bin, & Mohd Matore, M. E. E. Bin. (2020). Rating performance among raters of different experience
through multi-facet rasch measurement (MFRM) model. Journal of Measurement and Evaluation in Education and
Psychology, 11(2), 147–162. https://doi.org/10.21031/epod.662964

Mohd Noh, M. F., & Mohd Matore, M. E. E. (2022). Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis. Frontiers in Psychology, 13, 1-13 https://doi.org/10.3389/fpsyg.2022.941084

O’Grady, S. (2019). The impact of pre-task planning on speaking test performance for English-medium university
admission. Language Testing, 36(4), 505–526. https://doi.org/10.1177/0265532219826604

Ovilia, R. (2019). The Relationship of Topic Familiarity and Listening Comprehension. 276(Icoelt 2018), 182–186. https://doi.org/10.2991/icoelt-18.2019.26

Qiu, X. (2019). Functions of oral monologic tasks: Effects of topic familiarity on L2 speaking performance. Language Teaching Research. 24(6), 745-764. https://doi.org/10.1177/1362168819829021

Qiu, X., & Lo, Y. Y. (2016). Content familiarity, task repetition and Chinese EFL learners’ engagement in second language use. Language Teaching Research, 21(6), 681–698. https://doi.org/10.1177/1362168816684368

Rethinasamy, S., & Chuah, K. M. (2011). The Malaysian university English test (MUET) and its use for placement purposes: A predictive validity study. Electronic Journal of Foreign Language Teaching, 8(2), 234–245. https://doi.org/10.2139/ssrn.2146007

Sims, M. E., Cox, T. L., Eckstein, G. T., Hartshorn, K. J., Wilcox, M. P., & Hart, J. M. (2020). Rubric Rating with MFRM versus Randomly Distributed Comparative Judgment: A Comparison of Two Approaches to Second-Language Writing Assessment. Educational Measurement: Issues and Practice, 39(4), 30–40. https://doi.org/10.1111/EMIP.12329

Skehan, P. (2014). Processing Perspectives on Task Performance. John Benjamins Publishing Company.

Tahsildar, M. N., & Yusoff, Z. S. (2014). Investigating L2 students’ listening anxiety: A survey at a Malaysian university. International Journal of Language Education and Applied Linguistics, 1, 43–52. https://doi.org/https://doi.org/10.15282/ijleal.v1.418

Weir, C. J., & Wu, J. R. W. (2006). Establishing test form and individual task comparability: A case study of a semi-direct speaking test. Language Testing, 23(2), 167–197. https://doi.org/10.1191/0265532206lt326oa

Xiaolei, S., Ismail, L., Yurong, H., & Mengqi, W. (2023). Strategies of Content Knowledge Representation and EFL Learners’ English Writing Proficiency: Mediating Role of Critical Thinking Skills. Arab World English Journal, 14(3), 309–323. https://doi.org/https://dx.doi.org/10.24093/awej/vol14no3.19

Xu, J., & Qiu, X. (2021). Engaging L2 Learners in Information-gap Tasks: How Task Type and Topic Familiarity Affect Learner Engagement. RELC Journal, 0(0). https://doi.org/10.1177/00336882211061628

Facebook
Twitter
LinkedIn
Tumblr
Reddit
Email
StumbleUpon
Digg
Received: 07/11/2023  
Accepted: 11/28/2023 
Published: 12/15/2023
http://orcid.org/0000-0001-9838-2243
https://dx.doi.org/10.24093/awej/vol14no4.13   

Nurul Iman Ahmad Bukhari is teaching English at Universiti Malaysia Kelantan. She is also currently a PhD TESL candidate at Universiti Putra Malaysia.  Her research interests include ESL speaking performance and language assessment. http://orcid.org/0000-0001-9838-2243

Lilliati Ismail, Ph. D. is a Senior Lecturer at the Faculty of Educational Studies, Universiti Putra Malaysia. Her research interests include grammar instruction and task-based language teaching.  https://orcid.org/0000-0002-7977-7327.

Noor Lide Abu Kassim, Ph.D. is a Professor at the Kuliyyah of Education, International Islamic University Malaysia. Her research interests include Psychometrics and Education Evaluation.  http://orcid.org/0000-0003-2328-7406

Abu Bakar Razali, Ph.D. is an Associate Professor at the Faculty of Educational Studies, Universiti Putra Malaysia. His research interests include teaching of English as a second language (TESL), literacy and language education.  http://orcid.org/0000-0002-3181-1004

Nooreen Noordin, Ph.D. is an Associate Professor at the Faculty of Educational Studies, Universiti Putra Malaysia. Her research interests include content-based instruction, technology-enhanced language learning, learning styles, motivation, and best teaching practices in ESL.  http://orcid.org/0000-0002-4970-2682

Muhamad Firdaus Mohd Noh is an English primary school teacher. He is currently a PhD candidate at Universiti Kebangsaan Malaysia. His research interests include psychometrics and language assessment. http://orcid.org/0000-0002-5429-6789