Publications

2013
Hill, H. C., Gogolen, C., Litke, E., Humez, A., Blazar, D., Corey, D., Barmore, J., et al. (2013). Examining High and Low Value-Added Mathematics: Can Expert Observers Tell the Difference? In Association for Public Policy Analysis & Management Fall Research Conference . Washington, DC.Abstract

In this study, we use value-added scores and video data in order to mount an exploratory study of high- and low-VAM teachers' instruction. Specifically, we seek to answer two research questions: First, can expert observers of mathematics instruction distinguish between high- and low-VAM teachers solely by observing their instruction? Second, what instructional practices, if any, consistently characterize high but not low-VAM teacher classrooms? To answer these questions, we use data generated by 250 fourth- and fifth-grade math teachers and their students in four large public school districts.Preliminary analyses indicate that a teacher's value-added rank was often not obvious to this team of expert observers.

Download the conference paper
(2013). Graduate Exit Survey Toolkit . Strategic Data Project.Abstract

This toolkit provides useful resources for designing and rolling out a high school graduate exit survey, as well as effectively analyzing survey results in a school district. Anyone who is interested in implementing a high school exit survey, reworking a current exit survey, or effectively analyzing survey results in a school district can leverage this resource.

Learn more about the [Graduate Exit Survey].

Lynch, K., Chin, M., & Blazar, D. (2013). How Well Do Teacher Observations Predict Value-Added? Exploring Variability Across Districts. In Association for Public Policy Analysis & Management Fall Research Conference . Washington, DC.Abstract

In this study we ask: Do observational instruments predict teachers' value-added equally well across different state tests and district/state contexts? And, to what extent are differences in these correlations a function of the match between the observation instrument and tested content? We use data from the Gates Foundation-funded Measures of Effective Teaching (MET) Project(N=1,333) study of elementary and middle school teachers from six large public school districts,and from a smaller (N=250) study of fourth- and fifth-grade math teachers from four large public school districts. Early results indicate that estimates of the relationship between teachers' value-added scores and their observed classroom instructional quality differ considerably by district.

Download the conference paper
Hill, H. C., & Grossman, P. (2013). Learning from Teacher Observations: Challenges and Opportunities Posed by New Teacher Evaluation Systems. Harvard Educational Review.Abstract

In this article, Heather Hill and Pam Grossman discuss the current focus on using teacher observation instruments as part of new teacher evaluation systems being considered and implemented by states and districts. They argue that if these teacher observation instruments are to achieve the goal of supporting teachers in improving instructional practice, they must be subject-specific, involve content experts in the process of observation, and provide information that is both accurate and useful for teachers. They discuss the instruments themselves, raters and system design, and timing of and feedback from the observations. They conclude by outlining the challenges that policy makers face in designing observation systems that will work to improve instructional practice at scale.

Download full report
Chin, M., Hill, H., McGinn, D., Staiger, D., & Buckley, K. (2013). Using Validity Criteria to Enable Model Selection: An Exploratory Analysis. Association for Public Policy Analysis and Management Fall Research Conference.Abstract

In this paper, the authors propose that an important determinant of value-added model choice should be alignment with alternative indicators of teacher and teaching quality. Such alignment makes sense from a theoretical perspective because better alignment is thought to indicate more valid systems. To provide initial evidence on this issue, they first calculated value-added scores for all fourth and fifth grade teachers within four districts, then extracted scores for 160 intensively studied teachers.Initial analyses using a subset of alternative indicators suggest that alignment between value-added scores and alternative indicators differ by model, though not significantly.

Download working paper
2012
Hickman, J. J., Fu, J., & Hill, H. C. (2012). Technical Report: Creation and Dissemination of Upper-Elementary Mathematics Assessment Modules.Abstract

This document provides information on the grades 4 and 5 mathematics test modules developed as a joint Harvard/ETS venture from 2009 to 2012. Hundreds of items that were initially developed were then culled by a series of reviews, revisions, pilot tests, cognitive interviews, and field tests. 

ncte-assessments-tech-report.pdf
(2012). SDP College-Going Diagnostic for Gwinnett County Public Schools . Strategic Data Project.Abstract

Gwinnett County Public Schools worked with SDP to create the SDP College-Going Diagnostic for its district. The diagnostic is designed to identify potential areas for action to increase students’ levels of academic achievement, preparedness for college, and postsecondary attainment. It is also intended to demonstrate how districts can capitalize on existing data to understand its current performance, set future goals, and strategically plan responses.

Download full report
(2012). SDP Human Capital Diagnostic for Gwinnett County Public Schools . Strategic Data Project.Abstract

Gwinnett County Public Schools worked with SDP to produce the SDP Human Capital Diagnostic for its district. The diagnostic is designed to identify patterns of teacher effectiveness and areas for policy change that could leverage teacher effectiveness to improve student achievement. It is also intended to demonstrate how districts can capitalize on existing data to understand its current performance, set future goals, and strategically plan responses.

Download full report
(2012). SDP Human Capital Diagnostic for Los Angeles Unified School District . Strategic Data Project.Abstract

Los Angeles Unified School District (LAUSD) partnered with SDP to produce the SDP Human Capital Diagnostic for its district. The diagnostic is designed to identify patterns of teacher effectiveness and areas for policy change that could leverage teacher effectiveness to improve student achievement. It is also intended to demonstrate how districts can capitalize on existing data to understand its current performance, set future goals, and strategically plan responses.

Download full report
Jenkins, L., Wisdom, M., & Glover, S. (2012). Increasing College-Going Rates in Fulton County Schools: A Summer Intervention Based on the Strategic Use of Data. Purchase case study on Harvard Education Press websiteAbstract

This case study, published by Harvard Education Press, describes how to use data to challenge assumptions, reveal student needs, address these needs programmatically, and evaluate results. It shows a team of data specialists and educators working together, across institutional and departmental boundaries, to determine why some high school seniors who intend to go to college after graduation do not enroll in the fall. Together, they develop, implement, and evaluate a summer counseling intervention program called Summer PACE to ensure that more students enroll seamlessly in college.

Cascio, E. U., & Staiger, D. O. (2012). Knowledge, Tests, and Fadeout in Educational Interventions. Publisher's VersionAbstract

Educational interventions are often evaluated and compared on the basis of their impacts on test scores. Decades of research have produced two empirical regularities: interventions in later grades tend to have smaller effects than the same interventions in earlier grades, and the test score impacts of early educational interventions almost universally “fade out” over time. This paper explores whether these empirical regularities are an artifact of the common practice of rescaling test scores in terms of a student’s position in a widening distribution of knowledge. If a standard deviation in test scores in later grades translates into a larger difference in knowledge, an intervention’s effect on normalized test scores may fall even as its effect on knowledge does not. We evaluate this hypothesis by fitting a model of education production to correlations in test scores across grades and with college-going using both administrative and survey data. Our results imply that the variance in knowledge does indeed rise as children progress through school, but not enough for test score normalization to fully explain these empirical regularities.

Download full report
Hill, H. C., Charalambous, C. Y., Blazar, D., McGinn, D., Kraft, M. A., Beisiegel, M., Humez, A., et al. (2012). Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation. Educational Assessment , 17, 1-19.Abstract

Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to multiple sources of variation

Download full report
Tyler, J. H., Jacob, B. A., Dougherty, S. M., Hanson, H. J., Fullerton, J. B., & Herlihy, C. M. (2012). Are Practice-Based Teacher Evaluations and Teacher Effectiveness Linked in TNTP’s "Performance Assessment System (PAS)"? . Center for Education Policy Research at Harvard University.Abstract

The CEPR report, “Are Practice-Based Teacher Evaluations and Teacher Effectiveness Linked in TNTP’s Performance Assessment System (PAS)?” examines the evaluation system for first-year Louisiana teachers trained by TNTP, a national nonprofit organization focused on improving teacher performance.  The authors conclude that there is a modest positive relationship between teachers’ PAS scores and actual student achievement growth in math and reading.  The analysis also suggests that, with some technical improvements, the PAS could become an even better predictor of student academic outcomes.

Press Release Download full report
2011
(2011). SDP College-Going Diagnostic for Fulton County Schools . Strategic Data Project.Abstract

Fulton County Schools (FCS) partnered with SDP to produce the SDP College-Going and Human Capital Diagnostic for its district. The diagnostics are meant to demonstrate how districts can capitalize on existing data to understand its current performance, set future goals, and strategically plan responses.  The College-Going Diagnostic report illuminates students’ enrollment over time and compares these patterns across a variety of student characteristics and academic experiences.  The Human Capital Diagnostic report investigates teacher effectiveness with the intention of informing district leaders about patterns of teacher effectiveness and identifying areas for policy change that could leverage teacher effectiveness to improve student achievement.

Fulton County Schools College-Going Diagnostic Report Fulton County Schools College-Going Board of Education PowerPoint Presentation
(2011). SDP Human Capital Diagnostic for Fulton County Schools . Strategic Data Project.Abstract

Fulton County Schools (FCS) partnered with SDP to produce the SDP College-Going and Human Capital Diagnostic for its district. The diagnostics are meant to demonstrate how districts can capitalize on existing data to understand its current performance, set future goals, and strategically plan responses.  The College-Going Diagnostic report illuminates students’ enrollment over time and compares these patterns across a variety of student characteristics and academic experiences.  The Human Capital Diagnostic report investigates teacher effectiveness with the intention of informing district leaders about patterns of teacher effectiveness and identifying areas for policy change that could leverage teacher effectiveness to improve student achievement.

Download full report
(2011). SDP College-Going Diagnostic for Fort Worth Independent School District . Strategic Data Project.Abstract

Fort Worth Independent School District (FWISD) collaborated with SDP to create the SDP College-Going Diagnostic to examine the district’s college-going enrollment and persistence rates.  The diagnostic is designed to identify potential areas for action to increase students’ levels of academic achievement, preparedness for college, and postsecondary attainment.

Download full report
Kane, T. J., Jacob, B., Rockoff, J., & Staiger, D. O. (2011). Can You Recognize an Effective Teacher When You Recruit One? Association for Education Finance and Policy , 6 (1), 43-74. Publisher's VersionAbstract

The authors administered an in-depth survey to new math teachers in New York City and collected information on a number of non-traditional predictors of effectiveness: teaching specific content knowledge, cognitive ability, personality traits, feelings of self-efficacy, and scores on a commercially available teacher selection instrument. They find that a number of these predictors have statistically and economically significant relationships with student and teacher outcomes. The authors conclude that, while there may be no single factor that can predict success in teaching, using a broad set of measures can help schools improve the quality of their teachers.

Read the full report
Papay, J., West, M., Fullerton, J., & Kane, T. (2011). Does Practice-Based Teacher Preparation Increase Student Achievement? Early Evidence from the Boston Teacher Residency.Abstract

Center researchers John Papay, Martin West, Jon Fullerton, and Thomas Kane investigate the effectiveness of the Boston Teacher Residency (BTR) in their working paper Does Practice-Based Teacher Preparation Increase Student Achievement? Early Evidence from the Boston Teacher Residency.  BTR is an innovative practice-based preparation program in which candidates work alongside a mentor teacher for a year before becoming a teacher of record in Boston Public Schools.

Download full report
Taylor, E. S., & Tyler, J. H. (2011). The Effect of Evaluation on Performance: Evidence from Longitudinal Student Achievement Data of Mid-career Teachers. Publisher's VersionAbstract

The effect of evaluation on employee performance is traditionally studied in the context of the principal-agent problem. Evaluation can, however, also be characterized as an investment in the evaluated employee’s human capital. We study a sample of mid-career public school teachers where we can consider these two types of evaluation effect separately. Employee evaluation is a particularly salient topic in public schools where teacher effectiveness varies substantially and where teacher evaluation itself is increasingly a focus of public policy proposals. We find evidence that a quality classroom-observation-based evaluation and performance measures can improve mid-career teacher performance both during the period of evaluation, consistent with the traditional predictions; and in subsequent years, consistent with human capital investment. However the estimated improvements during evaluation are less precise. Additionally, the effects sizes represent a substantial gain in welfare given the program’s costs.

Download full report Download summary
Kane, T. J., Taylor, E., Tyler, J., & Wooten, A. (2011). Identifying Effective Classroom Practices Using Student Achievement Data. The Journal of Human Resources , 46 (3), 587-613.Abstract

This paper combines information from classroom-based observations and measures of teachers’ ability to improve student achievement as a step toward addressing the challenge of identifying effective teachers and teaching practices. The authors find that classroom-based measures of teaching effectiveness are related in substantial ways to student achievement growth. The authors conclude that the results point to the promise of teacher evaluation systems that would use information from both classroom observations and student test scores to identify effective teachers. Information on the types of practices that are most effective at raising achievement is also highlighted.

Download full report

Pages