Library

Quantitative Analysis of Indicators on the RTOP and ITC Observation Instruments

Abstract

Classroom observation is an important component of mathematics and science teacher professional development programs and other educational evaluation activities. In this paper, the authors extend their earlier qualitative examination and comparison of two popular classroom observation tools (Horizon Research, Inc.'s Inside the Classroom Observation and Analytic Protocol (ITC) and Sawada et al.'s Reform Teacher Observation Protocol (RTOP)) with a quantitative analysis. The instruments substantially are based on comparable assumptions, foundational philosophies and domains of interest, and appear to be used with an expectation that they yield similar results, not to mention that their domains adhere internally. A single well-trained educator observed 21 teachers from a Mathematics and Science Partnership over the course of two years. Pearson Correlation Coefficient analysis was applied to items across and within instruments. Because of the relatively small number of teachers and the use of one rater, the authors applied a relatively strict interpretation (.75) of high correlations. While some items correlated as expected within and between instruments based on domain and item construction, many items lacked matches, including those with an apparently similar focus. RTOP items showed greater alignment than did ITC items. Among summative "synthesis" domain ratings in the ITC, none of the four aligned with any of the specific ITC item indicators within their own domain categories. Subtle differences in wording and implicit differences in overall focus especially appear to restrict both internal matches and matches between superficially comparable items across the two instruments. These findings complicate the interpretation of observational results and challenge assumptions that the instruments are interchangeable or necessarily internally consistent. Additional research and development of observation instruments is needed, and users of existing instruments must carefully assess their own needs and understandings before attempting to draw conclusions about classroom practice based on them.