Opt-Out? Part 1 of 4: The Meaning of Measurement

There's a movement sweeping the nation right now. Parents and students are choosing to "opt-out" of required standardized testing from New York City to Seattle. The "opt-out" movement is only the latest manifestation of resistance to standardized testing, but it is one that has gained an incredible amount of momentum in a short amount of time. Close to home, every single 11th grade student at Nathan Hale High School, in Seattle, boycotted the recent Smarter Balanced Assessment test. Many educators who I respect are jumping on this movement, going so far as to call it an "Education Spring". Meanwhile, others are sitting on the sidelines with me, questioning whether this is the right thing to support or not. This issue has been particularly thorny for me. I'm an engineer by training, and now a teacher by trade. I can see the merits of all sides of the standardized testing debate. Slowly but surely, I figured, I can resolve this confusion and come out with a clear opinion on standardized testing.

I've written a few draft blog posts already. I've realized that the more I try and nail down my opinion on standardized testing, the more it seems to elude me. Houman Harouni talks about this tendecy towards confusion in his essay, "A Question of Silence": Why We Don't Read Or Write About Education. He says that "It’s a bloody mess trying to distinguish a trivial discourse on education from a significant one," in part because we choose not to acknowledge that we often have differing views on the whole purpose of education. Testing and measurement in education is a monumentally important and complex issue. It's one that defies easy answers and is very much rooted in these big theoretical questions Harouni nods to.

With Harouni's words ringing in my ears, instead of trying to lay out a nice and organized opinion, I'll try and open up some deep discussions about standardized testing. By doing so, I hope to describe what I currently believe and also what I'm still trying to figure out. Also, I hope that the complexity of these questions becomes clear and we'll realize together that many issues in education are inextricably linked. Over the next week or two I'll launch into the following four questions:

  1. Do standardized tests measure things that are meaningful?
  2. Is standardized testing a social justice issue?
  3. Are standardized tests a tool used by the rich to try and take over public education?
  4. Should your student opt-out of standardized tests?

Part 1: Do standardized tests measure things that are meaningful?

"The folks who create standardized achievement tests are terrifically talented. What they are trying to do is to create assessment tools that permit someone to make a valid inference about the knowledge and/or skills that a given student possesses in a particular content area. More precisely, that inference is to be norm-referenced so that a student's relative knowledge and/or skills can be compared with those possessed by a national sample of students of the same age or grade level."

W. James Popham http://www.ascd.org/publications/educational-leadership/mar99/vol56/num06/Why-Standardized-Tests-Don%27t-Measure-Educational-Quality.aspx/…

"I do not believe that standardized tests ... have any value whatsoever, for anybody except those who make money from them ... The acts of thinking, learning, and knowing, are not objects that can be replaced with abstractions about thinking, learning, and knowing. In that specific but crucial sense, all school test data are fake."

Ian Altman http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/08/14/seven-things-teachers-are-sick-of-hearing-from-school-reformers//…

"What researchers have long known is that what standardized tests measure above all else is a student's access to resources. The most damning truth about standardized tests is that they are a better indicator of a student's zip code than a student's aptitude."

Jesse Hagiopan http://www.amazon.com/More-Than-Score-Uprising-High-Stakes/dp/1608463923//…

Norm-referenced standardized tests, a common target for opt-out supporters, are designed to measure how well students can demonstrate their knowledge in a particular content area. Further, they intentionally place students on a percentile spectrum, providing information about how students compare to each other. This information is often misused by schools and governments, but at face value this information can be very meaningful. No test designer would ever claim that these tests actually capture the essence of learning. Rather, they are designed to estimate learning via an imperfect measurement. The notion that standardized test data is "fake" because it fails to completely represent learning is both completely true and completely irrelevant. We can never accurately measure holistic education, but that doesn't mean we shouldn't measure anything. We should seek to measure things that might be related to education, being careful not to rush our analysis of what causes what. Estimating what students know allows us to take concrete steps towards improving education for each of them.

The notion that standardized test data is "fake" because it fails to completely represent learning is both completely true and completely irrelevant.

In the 1960's, NASA engineers pioneered the use of an estimation technique called the Kalman Filter to guide the Apollo capsules to the moon. Previously, engineers were completely stumped by the crushing reality of something called Heisenburg's uncertainty principle, which meant that NASA could not measure both the position and the velocity of a spaceship at one time. Essentially, the closer that you get to quantifying something, the more you interfere with the thing that you're trying to measure.

Because of this, charting a course to the moon was all but impossible until Kalman realized that this problem really didn't matter. When it comes to measuring things that are impossible to measure, what matters is that we estimate them well. Kalman's technique was a breakthrough in estimation theory that guaranteed that Apollo's onboard computers could estimate their position and velocity with increasing accuracy as they travelled further away from Earth. The remarkable implications of this breakthrogh are still changing the face of science today. As a professor told me in college, "the Kalman Filter is the only reason we ever reached the moon." I don't think he was exaggerating.

Standardized tests are statistically relevant and informative, both for individual students and for groups of students. It is easy to develop an over-reliance on test scores, and in many cases, I think that we have done this. Fixing this problem does not mean abandoning test scores, though. We should try to be smart about when we collect data and how we use it. The Apollo missions found a way to estimate something that was immeasurable in a meaningful way. Because of it, they reached the moon.

When it comes to measuring things that are impossible to measure, what matters is that we estimate them well.

Those in the education sphere who advocate abandoning standardized tests are basically saying we should fly blind. This is an over-reaction and represents backwards thinking when it comes to improving education. The NAACP and 11 other civil rights organizations agree: "the anti-testing efforts that appear to be growing in states across the nation, like in Colorado and New York, would sabotage important data and rob us of the right to know how our students are faring. When parents ‘opt out’ of tests – even when out of protest for legitimate concerns – they’re not only making a choice for their own child, they’re inadvertently making a choice to undermine efforts to improve schools for every child."

Hagiopan and Altman hint at some of these "legitimate concerns." These concerns are inseparable from the institution of standardized testing and must be attended to. However, their presence does not make testing meaningless. Specifically, Hagiopan seems to suggest that since zip code predicts test scores, the tests are not meaningful. Technically, the correlation between income and performance on standardized tests says nothing about whether those tests estimate academic performance. It also doesn't mean that this correlation is fixed. Nate Bowling asks, "if you really believe student outcomes are immutable and predetermined by geography, why do you even bother rolling out bed in the morning?" The correlation between income and performance does raise some much larger questions, though. What knowledge do standardized tests value and not value? What messages do poor scores on a standardized test send to individual students or families from different backgrounds? These are essential questions to consider, ones that I'll dig into in my next post.

For now, I believe that standardized tests do estimate things that are meaningful for education. It takes great discernment, insight, and reflection when using the results of standardized tests to change education, but that is (or should be) the responsibility of professionals in the education field.

Moving forward, the elephant in the room is the fact that it is impossible to separate the details of collecting data from the details of how it is used. I've noticed that most people who oppose standardized testing do so because of the ways that the data is used. This deserves exploration. Next, I'll dive into this intersection of civil rights, social justice, and testing as I continue to try and figure out where I lie in this complicated debate. Stay tuned...

Read Part 2...

comments powered by Disqus