DEVELOPMENT AND VALIDATION OF AN AI LITERACY ASSESSMENT TOOL FOR HIGHER EDUCATION FACULTY IN CHINA: INSIGHTS FROM UNESCO’S AI-CFT FRAMEWORK

Author: Juhua Dou

ABSTRACT

As artificial intelligence (AI) transforms education globally, assessing teacher AI literacy has become critical—particularly in large-scale, policy-driven systems like China’s. This study develops and validates a contextually grounded assessment instrument for higher education faculty in China, based on UNESCO’s Artificial Intelligence Competency Framework for Teachers (AI-CFT). Through a two-phase validation process—including pilot testing (N = 40) and scale refinement via item analysis, exploratory factor analysis (EFA), and reliability testing—the initial 30-item pool was reduced to a 21-item scale that faithfully reflects the framework’s five dimensions and three proficiency levels. The final instrument demonstrates strong internal consistency (α=0.914) and coherent factor structure, offering a psychometrically sound tool for evaluating AI literacy in non-Western, high-digitalization educational contexts. This work not only advances the cross-cultural operationalization of UNESCO’s AI-CFT but also provides a validated foundation for teacher development, policy implementation, and comparative research on AI readiness in higher education.

Keywords: Teacher AI Literacy; Assessment Instrument; UNESCO AI-CFT Framework; Scale Development; Chinese Higher Education

REFERENCES

  1. Central Committee of the Communist Party of China, & State Council of the People’s Republic of China. (2025, January 19). Education power construction plan outline (2024–2035). https://www.gov.cn/zhengce/202501/content_6999913.htm
  2. (2023, June 1). UNESCO survey: Less than 10% of schools and universities have formal guidance on AI. https://www.unesco.org/en/articles/unesco-survey-less-10-schools-and-universities-have-formal-guidance-ai
  3. Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–16).
  4. Cukurova, M., & Miao, F. (2024). AI competency framework for teachers. UNESCO Publishing.
  5. Kočková, P., Kiliánová, K., Slepanková, M., et al. (2024). AI literacy in teacher education in the Czech Republic. In Proceedings of the 23rd European Conference on e-Learning (pp. 321–328). Academic Conferences International.
  6. Zhang, H., Lee, I., & Moore, K. (2024). An effectiveness study of teacher-led AI literacy curriculum in K-12 classrooms. Proceedings of the AAAI Conference on Artificial Intelligence, 38(21), 23318–23325.
  7. Ding, L., Kim, S., & Allday, R. A. (2024). Development of an AI literacy assessment for non-technical individuals: What do teachers know? Contemporary Educational Technology, 16(3), ep512.
  8. Karataş, F., & Yüce, E. (2024). AI and the future of teaching: Preservice teachers’ reflections on the use of artificial intelligence in open and distributed learning. International Review of Research in Open and Distributed Learning, 25(3), 304–325.
  9. Guo, Y., & Yu, H. (2023). Exploration of education transformation and teacher literacy in the age of artificial intelligence. In 2023 5th International Workshop on Artificial Intelligence and Education (WAIE) (pp. 38–42). IEEE.
  10. Yan, Z., Fu, J., Zhu, Y., et al. (2020). Integrating artificial intelligence into subject pedagogical knowledge (AI-TPACK): Connotation, teaching practice, and future issues. Distance Education Journal, 38(5), 23–34.
  11. Miao, F. (2024). Autonomous AI application based on teachers’ rights and interests: An interpretation of UNESCO’s AI competency framework for teachers. Open Education Research, 30(5), 4–16.
  12. Sun, L., & Xu, F. (2025). Practical pathways for AI-enhanced teacher development: An analysis based on UNESCO’s AI competency framework for teachers. Journal of Gansu Open University, 1, 1–5.
  13. Zhou, J. (2020). Questionnaire data analysis: Six analytical approaches using SPSS (2nd ed.). Publishing House of Electronics Industry.
  14. Fornell, C., & Larcker, D. F. (1981). Structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.