Going Beyond 1, 2, 3: New Data Systems and Quality Assessments for Oregon’s Early Childhood Programs

“This work gives us the big picture. It tells us what the State of Oregon is doing to help support its young children and the providers who care for them. It helps us understand if the whole of those services are provided efficiently and equitably, and whether we are meeting families’ needs.” - SDP Fellowship Alumnus Dr. Dylan Antovich 

"What are the goals and purposes of preschool? The very idea of quality and what aspects of preschool are most important can be different for different families. For this reason, having a diversity of high-quality preschool program options is important to successfully serve Oregon's young children and families.” - SDP Fellowship Alumnus Dr. Jaiya Choles 

Like building blocks and puzzle pieces, high-quality early-learning programs come in many shapes and sizes. But their benefits are universal. Strong early-learning experiences nurture a fast-growing brain, build a young child’s knowledge and social skills, and support working families by providing safe, enriching care. 

The State of Oregon has provided publicly funded preschool for  families with lower incomes for nearly 40 years, and a 2019 state law substantially expanded those efforts. A 2021 state law also established a new state agency with a complicated mission: the Department of Early Learning and Care (DELC), which brings a variety of subsidized childcare programs under a single roof. The department’s new data and analytics unit is tasked with supporting DELC’s diverse data needs, including providing data system strategy and establishing preschool quality assessments. 

Two Strategic Data Project (SDP) fellows at DELC took on this work: Dr. Dylan Antovich, Senior Data Analyst, focused on building a foundation for new data-management systems and practices to capture information from a variety of childcare programs and funding agencies. Dr. Jaiya Choles, Senior Data Analyst, worked on developing methods for collecting and analyzing preschool quality assessment data to track and support program improvement over time. 

“These are systems that are much more disparate than the K–12 system is,” said Choles. “Early childhood programs pose a unique challenge when it comes to measurement and data.” 

Establishing an Early Childhood Integrated Data System 

How many Oregon children are supported by publicly funded programs? Which programs have the biggest benefits to families over the long run? 

The answers are not in any one office or agency—Oregon supports a broad variety of programs for families with children ages 0–5, including early care and education for preschoolers, infants and toddlers, and developmental assessments and therapies. The state also provides related supports for families with lower incomes, such as housing vouchers, cash assistance, and healthcare. Federally funded efforts like Head Start, as well as locally funded programs like Portland’s Community Care Initiative, are also part of the puzzle. 

“Oregon needs a view of all of the early-childhood work going on at the state level,” said Antovich. “We offer different services, but everything is siloed. Coordinating the data will allow us to understand whether things are running smoothly and demonstrate the value of these programs.” 

As an SDP Fellow, Antovich worked within his agency and with consultants to envision what building an integrated data system would entail. The project focused on two initial challenges: modernizing internal data management within DELC and connecting data across state agencies.  

In collaboration with internal staff and external vendor supports, Antovich conducted an inventory of available tools and data, interviewed data leaders in other states, and provided recommendations to improve DELC’s data stewardship. As a result, DELC has begun to implement a formal data governance framework and new central data platform. While connecting data across state agencies will take several more years, the department’s progress can serve as a foundation from which to build. 

Oregon is at the outset of this work, and Antovich’s recommendations and insights were greatly informed by conversations with peers in other states. He credited SDP with connecting him with peers and a community of practice through ECDataWorks, as well as SDP alums Jennifer Koester, the Director of Enterprise Data for the State of Delaware, and Logan Rupard, then at KYStats, a state office within Kentucky’s Education and Workforce Development Cabinet. 

“These are folks who had developed similar systems in their states,” he said. “Talking with them was a great resource.” 

Assessing Quality in State-Funded Preschool Classrooms 

The second SDP Fellow, Choles, focused on collecting new data to assess preschool quality, with the goal of informing decision-making, job-embedded professional learning, and technical assistance. This project, primarily funded by the Bill & Melinda Gates Foundation, faced data-based technical hurdles and significant logistical challenges during its first year of implementation, which resulted in a successful, but smaller sample than was planned for the year. 

Like many states, Oregon’s publicly funded preschool programs contract with a variety of providers whose classrooms may be in private homes, brick-and-mortar buildings, and public and private schools. The state also funds preschool through two different programs: Preschool Promise and Oregon Prenatal to Kindergarten. What sort of sampling methodology and classroom-based data could accommodate these different inputs and create datasets that reliably depict classroom quality? 

With her supervisor Dr. Alicia Miao, Data and Analytics Manager, Choles met with DELC staff and consultants from Mathematica to diagnose what provider- and classroom-based data collection efforts should include. That work yielded a two-year plan to conduct observations in approximately 1,000 classrooms at 484 preschool sites, which were identified by a stratified random sampling methodology created by Choles.  

For the site visits, the preschool provider can choose between one of two observation tools. The provider receives the scores and a high-level summary of strengths and areas for improvement. Eventually, this initiative can grow into a statewide data-collection system and new indicator of program quality.  

Choles also developed a data literacy guide for DELC staff, inspired by “a passion for data ethics” and DELC’s dual role in collecting and using data. 

“The data collected from a three-hour snapshot is best used for coaching or technical assistance,” she said. “If it was to be used to determine quality or make decisions about funding—that would require different data. I thought it was very important for DELC staff to have that understanding.” 

Lessons Learned 

In reflecting on their work as SDP Fellows, Antovich and Choles came to similar conclusions. First, launching new initiatives like these takes more time than anticipated, because change at the state level includes building new procedures, adjusting contract language, and communicating changes with staff and providers. 

Both fellows also recommended starting with a pilot before fully rolling out a new data-collection program, to train up contractors and set expectations with internal staff about the appropriate use of the new data. 

In addition, both fellows also noted that for projects to be successful, they must be aligned with the goals of system partners outside of the immediate project team and built to outlive changes in leadership. This starts with drafting clear statements of the project’s vision and strategy, so the purpose of the project is easily identified. It also calls for planning the project alongside the future partners in other state agencies. 

Finally, the success of creating a detailed dataset of Oregon’s early-childhood programs requires building a trusting partnership with program providers, Choles noted. 

“Systematically collecting data for quality-improvement purposes allows us to provide assistance to grantees,” she said. “We can help them do what they are required to do and give them the supports that they need.”  

SDP is grateful to the Bill & Melinda Gates Foundation for their continued funding support for these fellows and the Cohort 16 SDP Fellowship Early Education Working Group. 

This post was originally published on the Strategic Data Project website on May 29, 2024.