Dangerously Irrelevant

Diane Ravitch says:

[The] data-driven focus [of Houston Independent School District's Apollo Program] contains the seed of its own destruction. Talking about tests all the time, doing test prep all the time, making kids take tests that they are not relevant to them and that they are not prepared for. . . .

I was not surprised by the emotional and physical reactions of these kids as staff kept trying to get them and keep them in school. The kids keep saying that the learning is irrelevant. They keep saying that school is boring. They keep saying that no one understands them and their plight. Telling them, “No Excuses!” is disrespectful.

via http://dianeravitch.net/2013/05/17/pbs-in-houston-watch-the-faces-of-the-students

Mike Wiser at The Quad-City Times reported today on the controversy here in Iowa around connecting student test scores to teacher evaluations (aka ‘value-added modeling’ or ‘VAM’). Last week I shared the research and prevailing opinion of scholars supporting why this should not be done.

In the article, notes that ‘teacher accountability has to be be part of it, or it’s not reform.’ This is consonant with policymakers’ general willingness to ignore the rating volatility concerns associated with VAM. As Amrein-Beardsley, et al. (2013) noted:

Policymakers have come to accept VAM as an objective, reliable, and valid measure of teacher quality. At the same time, [they ignore] the technical and methodological issues.

There appears to be a blind faith by many legislators in the objectivity of VAM, even though the actual data show that there is extremely high volatility in teacher ratings from year to year. Somehow policymakers are able to dismiss that rating instability as unimportant, even though it has tremendous impacts on teachers’ lives and reputations and public faith in the educational system. When Teachers of the Year are being rated ‘unsatisfactory’ by VAM systems, parents are rightfully suspicious. When high-achieving schools are rated as ‘needing improvement’, the public rightfully suspects that something’s not right. It’s important to note that legislators are not asking other professions to accept evaluation schemes in which 30 to 50 percent (or more) of their ratings fluctuate widely and completely randomly.

Of greater concern to me, however, is the response of Tom Narak, lobbyist for the School Administrators of Iowa (SAI). SAI represents all of the principals and superintendents in the state and is supposed to be knowledgeable about educational research and policy. Yet Mr. Narak says about VAM, “Why wouldn’t you? It’s the way (evaluations) are going now.”

Well, Mr. Narak, here are a few big reasons why we wouldn’t:

• Because year-to-year ratings for teachers are randomly varying 30%, 40%, 50%, or even higher [Di Carlo; Economic Policy Institute; Baker; National Education Policy Center]. In other words, extremely high percentages of teachers’ evaluations have absolutely nothing to do with their actual performance. As lobbyist for the administrators responsible for evaluating teachers, this should be alarming to you, not dismissed out-of-hand. Do you want principals and superintendents to send the message to their teaching staffs that they don’t care if evaluations are fair?
• Because even when student test scores are averaged over 3 to 5 years, random variation in teacher ratings still results in over 25% to 48% of teachers being rated inaccurately [U.S. Department of Education; Di Carlo]. In other words, when it comes to rating instability, looking over a longer time frame helps some but not a lot.
• Because the National Research Council, the National Academy of Education, the American Educational Research Association, RAND, the Annenberg Institute for School Reform at Brown University, the National Education Policy Center, James Popham, Gerald Bracey, Robert Linn, and many, many other of our most-respected scholars and research organizations all have looked at the research and said vehemently that we shouldn’t. In short, it’s a “who’s who” of educational policy and research – the folks we have trusted to inform us on policy decisions – all unilaterally aligned against VAM systems because of their volatility and unfairness.
• Because when VAM systems are implemented, predictably ludicrous and harmful results occur. These policy decisions have real consequences for our teachers for whom we supposedly have such great respect.
• Because even if we could devise a fair VAM system (which right now no one seems to be able to do), research shows consistently that the contribution of teachers to overall student test scores is 10% to 15% at most. The rest is attributable to other school factors or non-school factors. Any VAM system that imputes greater teacher responsibility than that small percentage would be highly unethical.
• Because holding teachers ‘accountable’ for random variation and/or factors outside of their control violates both the equal protection and due process rights due teachers under the U.S. Constitution.

If Mr. Narak and SAI are going to take a policy position on teacher evaluation, they should be up on the research I cited last week. In fact, on April 21 I e-mailed Mr. Narak the research noted above. Apparently, like many legislators, he and SAI don’t seem to care that the teacher evaluation systems for which they’re expressing support are inherently unfair and probably illegal? Would they feel the same if we were talking about the principals and superintendents whom they represent?

“Dear principal, 33% of your year-to-year evaluation will be completely random. Even though what you did this year isn’t substantially different from what you did last year, you may end up being rated highly or you may be rated near the bottom. Despite the extreme rating instability, there will be real consequences for you depending on the results. Good luck.”

Our teachers deserve evaluation systems that are fair. If they’re not fair, they’re unethical. If they’re not fair, they’re illegal. And right now, despite their intuitive appeal and legislative popularity in certain circles, VAM systems are unable to meet the basic principle of fairness and thus should not be supported by SAI or any other knowledgeable educational organization or policymaker.

[I'll also note as an aside that some states are starting to talk about evaluating administrators based on student test scores. If we are rightfully concerned about volatility in teacher ratings, wait until we remove the connection to students one additional step and try to tie scores to administrators. In other words, SAI, be careful for what you advocate because the principals and superintendents you represent are next…]

Finally, I’ll close with a plea to Jason Glass, Director of the Iowa Department of Education (DE), to publicly release the research that he has which supposedly supports VAM. Over the past months Jason has said repeatedly that DE and the Governor were not advocating for VAM approaches. And yet, here at the end of the legislative session, we somehow find ourselves discussing VAM systems and both DE and the Governor are supporting them. Whatever research Jason has, it’s going to somehow have to address the concerns noted above. Given that leading scholars and our most respected educational research/policy organizations are familiar with and have summarized the literature base and yet still strongly advocate against VAM, I’m skeptical. But, hey, maybe he’s got a bunch of dispositive studies with which both I and they are unfamiliar…

—–

I recognize that this post likely is going to make me unpopular with SAI (and even more unpopular than I already am with DE), which I regret because I’ve had good relations with them for a long time. But when the weight of the evidence is overwhelmingly against the policy position for which they’re advocating, I can’t just sit by and say nothing, not when it has very real, negative consequences for Iowa educators. John Ewing, President of Math for America, notes:

Of course we should hold teachers accountable, but this does not mean we have to pretend that mathematical models can do something they cannot.

I’ll state emphatically that we absolutely, under any circumstances, shouldn’t pretend that mathematical imprecision in evaluative processes has no impact on teachers’ lives and the fairness of our educational systems.

As always, I await your thoughts…

Should teachers be evaluated by students’ standardized test scores? While that idea seems to make intuitive sense, my newest resource on value-added measures (VAM) highlights the rating volatility, legal issues, and other concerns that have led our most trusted assessment experts and educational research/policy organizations to vehemently advocate against evaluating teachers with student test scores:

• Value-added measures

As we make policy, our teachers deserve our thoughtful, informed consideration. I hope this resource is helpful to you.

Did you know that…

• Google Apps can help you ‘launch a stress-free implementation of the Common Core’s required technology components?’
• these 5 apps will help your kids meet the new Common Core standards?
• FASTT Math is ‘the only fact fluency program to extend beyond math fact fluency practice into more rigorous Common Core objectives?’
• ‘Great Books programs meet Common Core standards?’
• these student center activities are ‘aligned to the Common Core?’
• all of these Pinterest resources are ‘aligned with specific Common Core standards?’
• every one of McGraw-Hill’s ‘core math, reading, and language arts programs fully align with the Common Core?’
• you can watch video clips of ‘Common Core-aligned teaching and learning?’
• NWEA ‘has developed a set of MAP assessments aligned to the Common Core?’
• you can get access to ‘the world’s first and only on-demand, comprehensive training on the Common Core standards?’
• all of these software programs are ‘aligned to the Common Core,’ even Timez Attack?
• you can ‘become a Common Core Black Belt?’
• the new SimCity video game will ‘conform with the Common Core?’

As expected, with the advent of the Common Core we are seeing a lot of labeling and re-labeling of instructional materials, resources, and activities. Publishers are adding the Common Core designation to existing textbooks, resources, assessments, and professional development opportunities just as fast as they can. Educators are unpacking the Common Core and affirming to themselves that they’re already doing what the standards expect. Lots of Common Core hoopla. Lots of Common Core assurances. Lots of old educational wine in new Common Core bottles…

Plus, of course, lots of gratuitous Common Core labeling and hucksterism. Because if it’s not stamped ‘Common Core’ these days, hardly anyone’s going to look at it. 

We have the standards. And publishers’ criteria. And state and school district certification efforts. But we also have lots of confusion, including whether or not teachers are prepared or unprepared to implement the standards.

As we sort out that confusion – and as we work together to become better prepared for implementation of the Common Core juggernaut – we need to be critical consumers of both our own lessons and the vendor pitches that accompany the standards. Because if there’s anything that policy-level folks agree on, it’s usually that the Common Core is supposed to be different. Very different.

• If the math in a ‘Common Core aligned’ textbook looks like the math you experienced as a student, put it down.
• the traditional US mathematics curriculum must become substantially more coherent and focused
• “I’m scared of rewarding bullsh*t,” Coleman told me…. “I don’t think it’s costless at all.” By bullsh*t, Coleman means the sort of watered-down curriculum that has become the norm in many American classrooms. [statement censored by me so as to better enable passage through draconian school Internet filters]

Of course if we absorb the Common Core into what we’ve always done without substantially changing anything – and this is extremely likely given our history – then things won’t be different at all. We know from past experience that standards usually don’t change instruction much. Neither do they change the day-to-day learning experiences of most children. Implementation always trumps wishes. Regardless of the rhetoric accompanying the Common Core, our historically high rates of reform assimilation indicate that what kids do in school on a daily basis is unlikely to be very different in most places. As Richard Elmore notes,

Internal accountability precedes external accountability and is a precondition for any process of improvement.

What does it mean to you for things to be ‘Common Core aligned℠?’ [Although Common Core chief architect / circus barker David Coleman believes that "people really don't give a sh*t about what you feel or what you think", I do.] Perhaps more importantly, what are you and your fellow educators doing to avoid old wine in new bottles?

P.S. Never fear. This blog post is Common Core-aligned℠. See ELA-Literacy.CCRA.R.8.

Iowa high school student Jack Hostager says:

Are you going to do what makes you look good or work for a system that does what’s best for students? Are we going to test the daylights out of me to get our bar a little bigger than everyone else’s on some national assessment data graph, or are we going to move towards a system that rewards meaningful learning and develops critical, concerned, productive citizens?

The laws that are in fashion demand tightly constrained curricula and reams of accountability data. All the better if it requires quiz-bits of information, regurgitated at regular intervals and stored in vast computers. Performance metrics, of course, are invoked like talismans. Distant authorities crack the whip, demanding quantitative measures and a stark, single number to encapsulate the precise achievement level of every child.

We seem to think that education is a thing – like a vaccine – that can be designed from afar and simply injected into our children. But as the Irish poet, William Butler Yeats said, “Education is not the filling of a pail but the lighting of a fire.”

This year, as you consider new education laws, I ask you to consider the principle of Subsidiarity. Subsidiarity is the idea that a central authority should only perform those tasks which cannot be performed at a more immediate or local level. In other words, higher or more remote levels of government, like the state, should render assistance to local school districts, but always respect their primary jurisdiction and the dignity and freedom of teachers and students.

Subsidiarity is offended when distant authorities prescribe in minute detail what is taught, how it is taught and how it is to be measured. I would prefer to trust our teachers who are in the classroom each day, doing the real work – lighting fires in young minds.

California Governor Jerry Brown via http://gov.ca.gov/news.php?id=17906

Too often, teachers and administrators complain that they are overly constrained by what the state government expects and cannot deviate for fear of not covering the necessary material.  This statement is somewhat true, however through authentic learning experiences i.e. content-specific authentic texts, evaluating real world issues, community service projects, labs, students can build an understanding of the world that takes them past any standard the state has set for them.  In any perpetually high-achieving school regardless of any demographic data, they do not teach to the test, they teach above the test.  Students’ experiences and knowledge take care of the rest.

Kevin ‘Doc’ Dougherty via http://educationismylife.com/curiosity-does-not-equal-chaos

notice what’s going on in the video around the 2:00 mark: a whole stageful of musicians, ranging over at least a span of 60 years of age, takes up bows and dives into the music. No one’s tracking them in an FMS (Fiddle Management System); no one’s worried about the failure to capture and embed Shareable Audio Objects. People put time and effort into becoming better at an activity they find worthwhile.

Dave Ferguson via http://www.daveswhiteboard.com/archives/5314

Hat tip: Stephen Downes, http://www.downes.ca/cgi-bin/page.cgi?post=59600

Parents may not understand the test score from the state or national test; however, they do understand and want to know how their child’s performance compares to other children across the state and nation.

Terry Holliday via http://kyedcommissioner.blogspot.com/2012/11/taking-stock-of-unbridled-learning.html

Yet another indication that parents don’t care about what their kids actually know and can do, they just care about how they compare to others?

Today the Iowa Department of Education (DE) released a report on achievement levels in Iowa compared to other states. The report also focuses heavily on closing the significant achievement gaps that exist in our state. Here are some very quick reactions that I have to the report…

1. The emphasis on better meeting the learning needs of traditionally-underserved student populations is absolutely necessary. Educationally and otherwise, we often have neglected students of color, students in poverty, students who are English language learners, and students with disabilities.
2. It’s hard to argue with proposed educational solutions that are focused on instruction, proven effective, and scalable, but I think that there is an accompanying, unstated concern: How should we think about educational initiatives that need to occur but don’t have ‘significant bodies of evidence’ behind them yet? For example, we live in a digital world and we know that students need to be fluent with the technologically-transformed information spaces of our time. And yet the peer-reviewed research to support this move isn’t there yet. It’s just sort of common sense: all we have to do is look around and realize that this is a need. Given the lack of ‘research,’ however, does that mean we don’t do it?
3. I wish that the report’s initial framing of the issues focused on the substantial changes that are occurring in the ways that we learn, citizenship needs in an increasingly-complex democracy, and other concerns related to life success beyond just economy/workforce issues. The latter are definitely important, but preparing future employees is not schools’ primary societal function.
4. If we’re going to work on raising scores and closing achievement gaps, let’s do our best to focus on assessments that matter. Right now we seem to be concerned mostly about average scores on assessments of primarily lower-level thinking. It’s also worth noting that our own National Research Council has found that decades of test-based incentives have done nothing to improve student learning outcomes. In fact, high school exit exams as configured in many states actually decrease graduation rates without concurrent increases in achievement.
5. Despite the sturm und drang around Iowa’s NAEP scores, we must recognize that there are no objective criteria and/or research-based evidence behind the cut scores for the different NAEP proficiency levels. The cut scores are set by committee and thus are inherently political. The NAEP benchmarks have been vociferously criticized by the National Academy of Sciences, the Government Accounting Office, the National Academy of Education, and many, many others. The designers of NAEP freely admit that the cut scores and levels are arbitrary.
6. Is our concern merely about raising Iowa students’ academic performance levels or is it necessary that we also BEAT OTHER STATES AND NATIONS? The rhetoric that’s flying around about Iowa ‘slipping to the middle of the pack’ seems very concerned about the latter. It’s also worth noting that most of the countries to which we negatively compare Iowa also wouldn’t do very well on NAEP.
7. ‘Rapid iteration,’ ‘living in perpetual beta,’ and other ideas related to quickly trying things, getting feedback to see if they worked, and adjusting course accordingly are all extremely important, particularly in a rapidly-changing world. As such, Response to Intervention (RTI) is a great process, particularly if feedback loops are short in time. But the RTI process also traditionally has been deeply rooted in notions of low-level cognitive work. Terms like ‘progress monitoring’ and ‘data-based decision-making’ are typically employed by educators in service of factual recall and procedural knowledge regurgitation. Turning those ideas toward higher-order thinking outcomes is going to be a lot of work in most school systems.
8. We need to be careful that we don’t turn ‘fidelity of implementation’ and ‘best practices’ into cookie-cutter instructional recipes and/or scripted lessons (as has occurred in many districts across the country). The report says that we need to ‘eliminate variability in instruction.’ I understand the sentiment behind that phrase but we need to be very wary of simplistic, stupid solutions to this issue.
9. The underlying premise of the report (and its accompanying policy proposals that we’ll see in the near future) is that education is a system amenable to fairly mechanistic solutions: put in place the right inputs, processes, and feedback loops and we’ll get the desired outcomes. Classic systems theory stuff. Learning and teaching are inherently messy domains, however, that often defeat externally-imposed procedures and expectations. As other nations show, we can improve student learning outcomes with thoughtful, purposeful changes, but we should be prepared for a lot of messiness along the way.
10. There’s a difference between ‘differentiation’ (as proposed in the report’s description of RTI) and ‘personalization’: see McClaskey & Bray’s chart on this. Differentiation is good, but a move away from primarily teacher-directed learning environments also is needed.

Final thoughts

Will teacher quality initiatives, the Iowa Core, and better deployment of RTI improve student learning outcomes in Iowa? Probably, at least somewhat. Are we going to see massive shifts in student learning outcomes in Iowa as a result of these? Probably not. These are school-focused interventions promulgated by the state department of education, and they’re all likely to have some positive impact. But they’re not enough. The research is very clear that roughly 80% of student learning outcomes is a result of NON-school factors. If we’re truly concerned as Iowa citizens and policymakers about improving student learning outcomes and closing achievement gaps, we’ll pay attention to the 80%, not just the 20%, just as most other ‘higher-performing’ nations have done. That means looking beyond the Department of Education for solutions.

Take some time to read over the report. What are your reactions?