Rethinking the assessment of the quality of teacher reflection by validating an innovative vignette-based instrument and an analytic coding scheme

Abstract

This study introduces the development of a new, transferable, customizable, authentic, and analytic framework for addressing different dimensions of teacher reflection. Given the emphasis on teacher reflection as a socially situated practice spawned by cognitive and affective aspects, this study covers four dimensions (i.e., breadth, depth, dialog, and affectivity). Ultimately and importantly, this empirical study constructs and validates an innovative vignette-based instrument and its analytic coding scheme on the basis of this framework for assessing the quality of reflection of elementary science teachers. The instrument includes eight vignettes, which include various authentic science teaching contexts, followed by three open-ended questions for each vignette to prompt teacher reflection on (1) the issues raised in the vignette, (2) the actions that could be taken to solve the problems, and (3) teacher emotions concerning the problems encountered. The instrument is validated through content, construct, internal, and face validity; and audit trial and inter-coder reliability are used to ensure the reliability of the data and findings. In summary, the study recruits 81 elementary science teachers from 24 public elementary schools in urban and suburban schools in Ankara for two reasons: (1) to ensure that the vignettes are clear and authentic as intended and (2) to develop a coding scheme involving descriptions and indicators of behaviors for each dimension of reflection quality accordingly to assess the quality of teacher reflection. For data analysis, once statistical conclusion validity and inter-coder reliability are established and a high percentage of agreement is obtained for each dimension across all vignettes (a > 0.70), content validity is once again explored by assessing whether the guiding framework of the instrument capture teacher reflections. The results indicate that the instrument can be an effective tool for assessing the quality of reflection of elementary science teachers. The preliminary validity and reliability of the vignettes and the coding scheme suggest that the vignettes warrant further testing.