Over the past few years, the Shanker Institute has been tracking and analyzing reading legislation. After NAEP results were made public, colleagues and friends began asking for my take on the link between literacy policy and NAEP reading outcomes. While many experts in student assessment have written extensively about NAEP’s dos and don’ts —here’s a recent example — I wanted to offer my perspective because, as Morgan Polikoff wisely cautioned in ‘Friends Don’t Let Friends Misuse NAEP Data,’ we must use the data responsibly. I understand the eagerness to see policy efforts make a difference for students; however, expecting too much too soon can be misguided and may even sabotage good policy efforts.
First and foremost, NAEP scores provide extremely valuable information about how U.S. students perform in various subjects in any given year. Using NAEP to advocate for improving academic outcomes makes a lot of sense. However, NAEP cannot specifically tell us why students are where they are or what can be done to improve their performance. And yet, raw NAEP scores are routinely misused—even at the highest levels — in this manner.
Why Is This a Problem?
For starters, NAEP data are cross-sectional, meaning that the same group of students is not tracked over time, unlike longitudinal data. As a result, changes in sociodemographic factors — such as the prevalence of child poverty or the percentage of English learners — can impact NAEP results. If factors such as these vary from year to year, test scores differences may be capturing these shifts, rather than only changes in educational measures. The Urban Institute, for example, publishes adjusted NAEP scores that account for some of these considerations.
Second, we implicitly assume NAEP results are explained by so-called in-school factors, such as the quality of instruction students receive, or the curriculum teachers use. This makes sense to a point; however, there are likely additional out-of-school or non-educational factors at play as well. Think about the COVID-19 pandemic, or the growth of digital culture and its impact on reading.
Third, the COVID-19 pandemic not only affected the amount and quality of instruction during school closures but also appears to be reshaping other behaviors. For example, public school enrollment and attendance are down; while homeschooling and privatization are on the rise.
The bottom line, as my colleague Matt Di Carlo wrote some time ago, is “if you simply eyeball the data and then make causal arguments, you’re really just guessing.” We need sophisticated analyses to understand what may be causing NAEP changes. Yet, at precisely this moment, we face cuts to critical funding streams for education research and data collection (and here) and proposals to dismantle the Department of Education. “NAEP maintains its gold standard through a network of contracts and data sources. Protecting NAEP means protecting all of these, or there will be further cuts and delays, and we will be further in the dark,” shared Andrew Ho on Bluesky. Without a robust infrastructure to gather, analyze, and contextualize data, we jeopardize our ability to discern the factors that drive student performance.
Even if we had an ideal set of literacy policies that were well-funded and implemented for at least five years, and we used rigorous methods to confirm that these policies impact NAEP, aggregate scores would likely show only small increases — consistent with the generally moderate effects seen from most educational interventions, particularly in reading. Moreover, some experts argue that reading tests function as knowledge tests in disguise. Growing up in economically-disadvantaged environments can negatively impact children’s school readiness and language development. Language often serves as just the tip of the iceberg, signaling broader and deeper differences in children’s knowledge. Consequently, students from these backgrounds might underperform simply because they are less familiar with the concepts being assessed.
With all these in mind, these are three do’s and one don’t:
- DO use NAEP data to highlight the need for improved literacy. The case for better reading outcomes is strong and universally agreed upon. We don’t need to be alarmist or hyperbolic to compel action.
- DO advocate for policies that align with reading research. This, to me, is the biggie. The main reason to improve how reading is taught is to close the longstanding gap between what evidence on reading shows and what teachers learn in their preparation programs, in-service professional development and, indirectly, through the instructional materials they use to teach reading. Even if educational policies have modest effects — as they often do — don’t we want to advance the best knowledge and materials to help children learn to read?
- DO support well-designed studies that examine the impact of policies on outcomes. When asking questions about the efficacy of educational policies, turn to carefully done studies that control for what has been implemented and when, as well as other state and local factors. Be an advocate for this kind of research, too, and speak up when it is indiscriminately canceled.
- DON’T use raw NAEP scores to judge the effectiveness of reading policies. Doing so could lead to a scenario where insufficient progress is seen, and good policies are abandoned prematurely for reasons unrelated to their true effectiveness or lack thereof.
While it may be tempting to overlook all the above — as many do — the fact that “everyone else does it” does not make it right. Do we really need improper use of good data to make the case for evidence-based reading policies? I don’t think so. While teaching reading in greater alignment with the evidence base will not be a panacea even in the best-case scenario (done well, with resources and given sufficient time) but it is a step in the right direction that will undoubtedly help many students learn to read more efficiently and with less stress. And that is why we should continue to advocate for these policies.
View the original article and our Inspiration here