In 2011-12, Newark launched a set of educational reforms supported by $20 million gift. Using data from 2009 through 2016, we evaluate the change in Newark students’ achievement growth relative to similar students and schools elsewhere in New Jersey. We measure achievement growth using a “value-added” model, controlling for prior achievement, demographics and peer characteristics. By the fifth year of reform, Newark saw statistically significant gains in English and no significant change in math achievement growth. Perhaps due to the disruptive nature of the reforms, growth declined initially before rebounding in recent years. Aided by the closure of low value-added schools, much of the improvement was due to shifting enrollment from lower-to higher-growth district and charter schools. Shifting enrollment accounted for 62 percent of the improvement in English. In math, such shifts offset what would have been a decline in achievement growth.
Researchers have identified many characteristics of teachers and teaching that contribute to student outcomes. However, most studies investigate only a small number of these characteristics, likely underestimating the overall contribution. In this paper, we use a set of 28 teacher-level predictors drawn from multiple research traditions to explain teacher-level variation in student outcomes. These predictors collectively explain 28% of teacher-level variability in state standardized math test scores and 40% in a predictor-aligned math test. In addition, each individual predictor explains only a small, relatively unique portion of the total teacher-level variability. This first finding highlights the importance of choosing predictors and outcomes that are well aligned, and the second suggests that the phenomena underlying teacher effects is multidimensional.
The purpose of this study is to investigate three aspects of construct validity for the Mathematical Quality of Instruction classroom observation instrument: (1) the dimensionality of scores, (2) the generalizability of these scores across districts, and (3) the predictive validity of these scores in terms of student achievement.
As many states are slated to soon use scores derived from classroom observation instruments in high-stakes decisions, developers must cultivate methods for improving the functioning of these instruments. We show how multidimensional, multilevel item response theory models can yield information critical for improving the performance of observational instruments.
Education agencies are evaluating teachers using student achievement data. However, very little is known about the comparability of test-based or "value-added" metrics across districts and the extent to which they capture variability in classroom practices. Drawing on data from four urban districts, we find that teachers are categorized differently when compared within versus across districts. In addition, analyses of scores from two observation instruments, as well qualitative viewing of lesson videos identify stark differences in instructional practices across districts among teachers who receive similar within-district value-added rankings. Exploratory analyses suggest that these patterns are not explained by observable background characteristics of teachers and that factors beyond labor market sorting likely play a key role.
To make up for pandemic-related learning losses, many U.S. public school districts have increased enrollment in their summer school programs. We assess summer school as a strategy for COVID-19 learning recovery by tracking the academic progress of students who attended summer school in 2022 across eight districts serving 400,000 students. Based on students’ spring to fall progress, we find a positive impact for summer school on math test achievement (0.03 standard deviation, SD), but not on reading tests. These effects are predominantly driven by students in upper elementary grades. To put the results into perspective, if we assume that these districts have losses similar to those present at the end of the 2022–23 school year (i.e., approximately -0.2 SD), we estimate summer programming closed approximately 2% to 3% of the districts’ total learning losses in math, but none in reading.
We analyze data from approximately 7,800 school districts to describe variation in pandemic-related learning losses among communities and student subgroups. We attempt to understand mechanisms that led to learning losses, as well as explore how historical data from those districts can inform our expectations for how quickly districts will rebound from such losses. We show that learning losses during the pandemic were large and highly variable among communities. Similar to previous research, we find that losses were larger in lower-income and minority districts and in districts which remained remote or hybrid for longer periods during the 2020-21 school year. Among districts, the math learning loss per week of remote/hybrid instruction was larger in high-minority and high-poverty districts. Within districts, however, White students and non-economically disadvantaged students lost about the same amount of ground as Black, Hispanic and economically disadvantaged students. This suggests that the mechanisms driving losses operated at the district or community level, rather than household level. Several community-level characteristics were related to learning losses: broadband access, disruptions to social and economic activity, and trust in government institutions. However, no individual predictor provided strong explanatory power. Relative to historical years, losses during the pandemic were substantial, and an exploratory analysis of historical shocks to achievement suggests that the effects of the pandemic are likely to persist without continued concerted investments in student learning.
During the summer and fall of 2022, researchers at the Center for Education Policy Research (CEPR) at Harvard University conducted a series of interviews to explore district leaders’ willingness to participate in evaluation efforts for academic products and services. There were three primary research objectives:
To determine what information district decision-makers need, want, and would find helpful in the identification and adoption of interventions and learning technology products and to better understand when in the procurement and decision-making process this information could be most useful.
To determine what barriers exist for schools and districts to pilot and test interventions before rolling them out system-wide, and which evaluation designs districts would be most willing to implement.
To understand what model of engagement is most desirable for districts to participate in rigorous evaluation work, including barriers to participation.
This study engaged 16 district leaders across the country in qualitative interviews. District leaders participating in this analysis represented 13 unique states with regional representation in the South (5), West (4), Southwest (4), Midwest (2), and New England (1). In total, leaders from participating districts serve 1.2 million students with individual district sizes ranging from about 25,000 to 200,000 students. All the leaders interviewed served a critical role in their district’s decision-making processes for the procurement of academic products and services. Most participating leaders were either part of their district’s leadership cabinet, or directly reported to a cabinet member. These interviews probed how district leaders currently incorporate evidence of effectiveness into their purchasing of academic products and services (i.e., academic interventions and core curricular materials), what evidence they wished they had, how willing they would be to have their district generate its own evidence through program evaluations, and what structures could be most helpful to support their decision-making moving forward.
Following the interviews, the research team also analyzed the studies that vendors provided to school districts during the procurement process. The purpose of this analysis was to better understand how vendors report on the effectiveness of their product, and to what degree these reports align with rigorous research methodologies.
Below, we summarize our findings and their implications for developing a network of districts focused on using high-quality evidence to select academic products and services. In Appendix A, we attach a brief outline of what such a network might look like and a timeline for its development.
In the three decades before the pandemic, mean achievement of U.S. 8th graders in math rose by more than half a standard deviation on the National Assessment of Educational Progress (NAEP). Between 2019 and 2022, U.S. students had forfeited 40 percent of that rise. To anticipate the consequences of the recent decline, we investigate the past relationship between NAEP scores and students’ later life outcomes by year and state of birth. We find that a standard deviation improvement in a birth cohort’s 8th grade math achievement was associated with an 8 percent rise in income, as well as improved educational attainment and declines in teen motherhood, incarceration and arrest rates. If allowed to become permanent, our findings imply that the recent losses would represent a 1.6 percent decline in present value of lifetime earnings for the average K-12 student (or $19,400), totaling $900 billion for the 48 million students enrolled in public schools during the 2020-21 school year.
In this paper we examine academic recovery in 12 mid- to large-sized school districts across 10 states during the 2021–22 school year. Our findings highlight the challenges that recovery efforts faced during the 2021–22 school year. Although, on average, math and reading test score gains during the school year reached the pace of pre-pandemic school years, they were not accelerated beyond that pace. This is not surprising given that we found that districts struggled to implement recovery programs at the scale they had planned. In the districts where we had detailed data on student participation in academic interventions, we found that recovery efforts often fell short of original expectations for program scale, intensity of treatment, and impact. Interviews with a subsample of district leaders revealed several implementation challenges, including difficulty engaging targeted students consistently across schools, issues with staffing and limitations to staff capacity, challenges with scheduling, and limited engagement of parents as partners in recovery initiatives. Our findings on the pace and trajectory of recovery and the challenges of implementing recovery initiatives raise important questions about the scale of district recovery efforts.
Using testing data from 2.1 million students in 10,000 schools in 49 states (plus D.C.), we investigate the role of remote and hybrid instruction in widening gaps in achievement by race and school poverty. We find that remote instruction was a primary driver of widening achievement gaps. Math gaps did not widen in areas that remained in-person (although there was some widening in reading gaps in those areas). We estimate that high-poverty districts that went remote in 2020-21 will need to spend nearly all of their federal aid on academic recovery to help students recover from pandemic-related achievement losses.
Over the past two decades, education underwent a “big data” revolution as states began tracking individual student performance and interim assessments and educational software allowed for a greater granularity of data on students, teachers, and schools.
Despite this plethora of new data, considerable gaps in data on early childhood education, school spending, student program and intervention, and postsecondary outcomes remain.
Dissatisfaction with education data will never fully disappear due to technical gaps between what policymakers and researchers would like to measure and what can be measured, as well as normative disagreements about what data should be collected.
Policymakers should focus on closing the gaps they can while also recognizing the technical and normative constraints on educational measurement.
Given the major disruptions to students’ daily lives as well as the education field more generally caused by the COVID-19 pandemic, NCRERN was interested in learning how its partner districts navigated mandatory school closures and the shift to online learning, as well as identifying ways that NCRERN could support the short- and long-term needs of rural educators. Throughout April 2020, NCRERN staff conducted semistructured phone interviews with district officials and other leaders from 40 out of its 49 partner rural districts in Ohio and New York. The majority of interviews took place when schools were 3–5 weeks into shutdown. Notes from each interview were coded by two graduate research assistants to identify major themes that emerged from the conversations. Because interviews were semistructured, not all districts answered each question; as a result, counts should be interpreted with caution.
Like many other elements of the American economy, higher education is working to realize the potential of sophisticated data analytics to inform and transform how it operates. In August 2019, the Association for Institutional Research (AIR), EDUCAUSE (the association of campus information technology professionals), and the National Association of College and University Business Officers (NACUBO) released a joint statement with the provocative title “Analytics can save higher education. Really.” Its purpose was to inspire a sense of urgency and provide direction for higher education leaders to harness data as a strategic organizational asset. The statement features the following rationale for investment in data analytics:
“We strongly believe that using data to better understand our students and our own operations paves the way to developing new, innovative approaches for improved student recruiting, better student outcomes, greater institutional efficiency and cost-containment, and much more.”
However, progress has been uneven, with some state higher education agencies, university and college systems, and individual institutions leading the way while many others struggle to adapt. Why?
The Strategic Data Project (SDP) at the Center for Education Policy Research at Harvard University has a ten-year track record of developing data capacity in state and local PK-12 agencies and organizations and interviewed 40 leaders and analysts at 29 institutions of higher education and postsecondary organizations to explore their data needs to understand why some colleges and university systems are excelling in using data and others have yet to fully realize the potential of their data to inform strategic decisions that transform student success in school and the workforce.
Our key finding is that the missing link is not in the technical infrastructure but in human capacity. If higher education is to take advantage of data analytics to improve student outcomes and increase organizational effectiveness, it will have to find better ways to attract, train, and retain strategic data professionals who can inform policy and practice.
Teacher evaluation reform has been among the most controversial education reforms in recent years. It also is one of the costliest in terms of the time teachers and principals must spend on classroom observations. We conducted a randomized field trial at four sites to evaluate whether substituting teacher-collected videos for in-person observations could improve the value of teacher observations for teachers, administrators, or students. Relative to teachers in the control group who participated in standard in-person observations, teachers in the video-based treatment group reported that post-observation meetings were more “supportive” and they were more able to identify a specific practice they changed afterward. Treatment principals were able to shift their observation work to noninstructional times. The program also substantially increased teacher retention. Nevertheless, the intervention did not improve students’ academic achievement or self-reported classroom experiences, either in the year of the intervention or for the next cohort of students. Following from the literature on observation and feedback cycles in low-stakes settings, we hypothesize that to improve student outcomes schools may need to pair video feedback with more specific supports for desired changes in practice.
Aided by $200 million in private philanthropy, city and state leaders launched a major school reform effort in Newark, New Jersey, starting in the 2011–2012 school year. In a coinciding National Bureau of Economic Research (NBER) working paper, we assessed the impact of those reforms on student achievement growth, comparing students in Newark Public Schools (NPS) district and charter schools to students with similar prior achievement, similar demographics, and similar peers elsewhere in New Jersey. This report includes key findings.