The Challenge of Execution: District Leader Insights on Evaluating Academic Programs, Products, and Services

During the summer and fall of 2022, researchers at the Center for Education Policy Research (CEPR) at Harvard University conducted a series of interviews to explore district leaders’ willingness to participate in evaluation efforts for academic products and services. There were three primary research objectives:

1. To determine what information district decision-makers need, want, and would find helpful in the identification and adoption of interventions and learning technology products and to better understand when in the procurement and decision-making process this information could be most useful.

2. To determine what barriers exist for schools and districts to pilot and test interventions before rolling them out system-wide, and which evaluation designs districts would be most willing to implement.

3. To understand what model of engagement is most desirable for districts to participate in rigorous evaluation work, including barriers to participation.

This study engaged 16 district leaders across the country in qualitative interviews. District leaders participating in this analysis represented 13 unique states with regional representation in the South (5), West (4), Southwest (4), Midwest (2), and New England (1). In total, leaders from participating districts serve 1.2 million students with individual district sizes ranging from about 25,000 to 200,000 students. All the leaders interviewed served a critical role in their district’s decision-making processes for the procurement of academic products and services. Most participating leaders were either part of their district’s leadership cabinet, or directly reported to a cabinet member. These interviews probed how district leaders currently incorporate evidence of effectiveness into their purchasing of academic products and services (i.e., academic interventions and core curricular materials), what evidence they wished they had, how willing they would be to have their district generate its own evidence through program evaluations, and what structures could be most helpful to support their decision-making moving forward.

Following the interviews, the research team also analyzed the studies that vendors provided to school districts during the procurement process. The purpose of this analysis was to better understand how vendors report on the effectiveness of their product, and to what degree these reports align with rigorous research methodologies.

Below, we summarize our findings and their implications for developing a network of districts focused on using high-quality evidence to select academic products and services. In Appendix A, we attach a brief outline of what such a network might look like and a timeline for its development.