When it Comes to Students, What Are We Measuring?
Ph.D. candidate Lily An helps shape how we evaluate students with her research
Educators want to help students, and oftentimes that involves measuring them. The numbers stem from important questions: How much are they learning? How is their learning impacted by various educational methods and policies? But before answering those types of questions, says Lily An, we first have to ask another complex question: When it comes to understanding students, how do we best measure them to begin with?
“At the end of the day, we're all trying to help students and support student learning and student outcomes,” says An, a Ph.D. candidate in HGSE’s Education Policy and Program Evaluation concentration. “So we owe it to them to keep looking and keep digging and keep trying to improve our practice.”
An came to HGSE after a career in finance, where she worked in equity research and evaluated whether companies would meet their earnings projections. She had always wanted a career in education, though, and joined HGSE’s master’s program. Learning about psychometrics in Professor Andrew Ho’s Statistical and Psychometric Methods for Educational Measurement class, however, opened up a new area of research for An. Psychometrics applies psychology and statistics to develop measures for assessing attributes such as personality traits and attitudes. In education, it’s used to study how student abilities are measured.
“I majored in statistics in college, so I thought, ‘Wow, this actually combines all my interests: understanding education and statistics and value,’” says An. “And the way that we do that a lot of times in education is through testing, so I think it's really important to be thinking critically about what we are valuing. What are we measuring?”
As her research interests became more defined, An began to focus on how organizations decide what and how to measure their students, and what impact those educational policies have on student outcomes.
“I switched to a more expansive, external, and organizational view,” she says. “There are all these policies and institutions and systems that are affecting access and what people can do.”
After graduating from HGSE in 2018 with her master’s, she returned in 2020 to pursue her Ph.D. in education. With advisers Ho and Associate Professor Luke Miratrix, she’s currently wrapping up her dissertation and is expected to graduate in May 2025 having spent the last four-plus years focused on educational measurement, causal inference methods, and school accountability policies.
“The question I'm always asking is ‘How can educational policies and the methods of evaluating those policies be more attentive to the measures that are used?’” says An. “We know that measures are important. We know that evaluation is important, but they're not always talking to each other. So where can we highlight those opportunities and improve them?”
An has worked with a variety of different partners in her academic career, including with Kentucky when the state developed different classifications of school quality levels. She also spent the two years between her master’s and Ph.D. programs at Brown University as a research analyst working for Annenberg Institute Director John Papay, Ed.M.’05, Ed.D.’11. The research, studying the impact of high-stakes testing exit examinations like the 10th-grade Massachusetts Comprehensive Assessment System (MCAS), was an illuminating example, she says, of walking the line between a partner’s needs and the importance of scholarly practice.
An’s work with Papay was initially published in 2018, but a presentation of more recent work on the subject at an Askwith Education Forum/PIER Public Seminar on the MCAS ballot measure by Papay showed the impact that research has as it evolves.
“I was like, ‘I remember that graph,’” An recalls of Papay’s presentation last October. “‘I originally made that graph!’”
An credits her work at Annenberg with opening up another key part of her HGSE journey: a Partnering in Education Research (PIER) fellowship.
Continue reading at www.gse.harvard.edu
Written by Ryan Nagelhout