Backlash against progress monitoring

The benefits of ongoing progress monitoring of students on essential academic
skills (i.e., “power standards”) are well established. But, as Sherman Dorn
notes
, some school districts are starting to see a backlash from students and
parents against too much testing, even when it’s formative assessment designed
to increase student mastery of critical content.

Here is the comment I
left on Sherman’s blog post:

I think one of the problems is that many school systems just throw a bunch of
formative assessments up on the wall, hoping some of them will stick regarding
their curriculum standards and/or year-end assessments. For example, I just
spoke with a principal whose elementary school was doing SIX different reading
assessments with its kids. When pressed to explain WHY each assessment was used
(i.e., what information each assessment gave teachers that the other ones
didn’t), he was unable to articulate how the assessments were similar /
different from each other. If schools wish to avoid student and parent backlash
against too much testing, they need to have an aligned assessment strategy that
clearly outlines when and why each assessment is being used. If schools can’t do
this, they deserve all the backlash they get.

Does your school organization have an aligned assessment system? Can it
clearly articulate when, why, and for which students each formative assessment
is being used as well as how that assessment overlaps (or doesn’t) with other
assessments?

School data tutorials

[cross-posted at LeaderTalk]

Even when principals and teachers have access to data, they often aren’t sure what to do with it. That’s why CASTLE (okay, it was me!) created School Data Tutorials, a web site intended to help K-12 educators work with raw student and school data.

The tutorials on the web site highlight many of the Excel skills that are helpful when working with building- and district-level data. The tutorials are targeted at data managers, principals, guidance counselors, teachers, and other school personnel who have the responsibility for collecting, analyzing, and reporting K-12 performance data (which is just about everyone these days!). You will see that the tutorials are much like the ones created by Atomic Learning (they trained us!) but are focused on data-driven decision-making needs of educators rather than being generic.

Below are the four sets of tutorials we recommend for every teacher and administrator. If every educator knew how to do these four things, schools’ capacity to do some basic monitoring of student progress would be greatly enhanced. They take just over half an hour to watch once, and of course they can be viewed as many times as necessary to accomplish mastery.

Many more tutorials are available on the site, including instructions on how to make your own data collection templates. Let me know if you feel empowered after watching these!

Freeze panes

Sort

Filter

Conditional formatting

Note that you may need to turn off your browser’s pop-up blocker or install the latest Flash plug-in to view these tutorials. Happy viewing!

ROTW: Assessing students with disabilities

The latest installment of the Report of the Week (ROTW) comes from Education Sector:

Here’s a quote from the report:

[B]ecause the majority of special education students have disabilities that do not preclude them from reaching grade-level standards there doesn’t seem to be a need to rollback NCLB’s accountability measures for students with disabilities. To the contrary, doing so could hurt such students by reducing schools’ responsibility for ensuring that they are taught to high standards.

Happy reading!

Interview with Phala Daniel

I have had the pleasure of working with the Chicago Public Schools (CPS) on several occasions over the past few years, primarily in conjunction with its Principal Technology Leadership Institute (PTLI), which is a collaborative venture with the International Society for Technology in Education (ISTE). I really enjoy the CPS folks. They face numerous challenges but are some of the most dedicated educators I know.

Some of the principals participating in the data-driven decision-making strand of the PTLI (which I helped design) had some questions that they wanted me to answer about data-driven leadership, so today I had a very pleasant conversation with Phala Daniel, who works in CPS’ eLearning office along with Gerry Beimler (who is one of our CASTLE graduate certificate students). I recorded our 37-minute discussion and present it for your listening pleasure:

FYI, here’s one example of a protocol for a data-driven team meeting (as mentioned in the conversation). Hope you enjoy the podcast. Let me know what you think!

Gapminder

I have seen the future of data presentation and it is Gapminder. Visit Gapminder and click on

  • the Play button (bottom left)
  • the geographic regions (top right)
  • country names (right)
  • the magnifying glass (bottom right of chart)
  • the Map tab (top left)

Also use the three sliders at the bottom (under Income per capita in international dollars and above and below Population). Notice also that merely hovering your mouse over different parts of the graphs highlights various information aspects. Finally, you can change both the x and y axes by clicking on the dropdown box triangles for Income per capita in international dollars (right of label) and Life expectancy, years (top of label).

I had seen the video of Gapminder founder Hans Rosling at TED Talks, but it’s fun to get to play with the software a little bit. How great would it be to have this tool for K-12 data-driven decision-making?

Wow.

Put your hands up and move away from the textbook

Put your hands up and move away from the textbook

00podcast16x16_4Listen to this post!

Dan Meyer is a dynamic young math teacher for the San Lorenzo Valley Unified School District. He totally gets it when it comes to assessment. Here are four great posts, the first three of which I wish I could get every teacher and administrator in Minnesota to read:

Thanks, Chris, for introducing me to Dan’s blog. You called him a ‘must-read.’ I think you’re right.

Not enough time to be data-driven

I do a lot of work with schools on data-driven accountability issues. Before you immediately decide that I’m just another data huckster, I’ll point out now that my work with schools focuses on good ongoing, formative assessment for student progress monitoring purposes rather than on the stupid yearly summative autopsy data that most schools are spending WAY too much time on. Intelligent use of progress monitoring data related to key academic and other educational goals has been shown time and time again, in both high-quality research studies and in tens of thousands of schools and classrooms across the country, to have significant impacts on student learning outcomes. The most common complaint that I hear from teachers, however, is that they’re already pressed – they don’t have time to add another thing to their plate. Balderdash. Here are a few things that teachers can get rid of that will free up some valuable time.

Teachers work extremely hard. They’re some of the most caring, dedicated people I know. But, like most of us, they often don’t use the time that they have very effectively (or others don’t use their time very intelligently). If we truly care about student learning, we should be taking a critical look at teachers’ precious time and try to eliminate many of our low-yield practices.

  • How about coloring? As Mike Schmoker points out so eloquently, students in elementary schools spend a LOT of time coloring. Couldn’t teachers do a little less of this and spend some of that time doing quick, timely, useful assessments of student learning and/or individualized instruction for struggling students?
  • Cursive writing, anyone? Does anyone think this will be a needed skill in the digital future? How much time and energy do teachers and students still spend on this?
  • One-shot, single day staff development sessions are almost always completely useless if our desired goal is to facilitate changes in teacher practice. What is remarkable is the persistence of this practice despite everything that we know about adult learners and effective professional development. Couldn’t we use this time for teachers to collaboratively examine student learning progress instead?
  • As Alfie Kohn reminds us, and despite decades of researchers trying, there is no evidence to support that homework in the elementary grades has any positive benefit whatsoever on student achievement, study skills, greater responsibility by students, etc. Teachers and students spend a lot of time processing homework – time that could be better spent elsewhere.
  • Watching entire movies in class instead of short video clips. Enough said.

You get my drift. The list probably could on for a while – each of you can think of other things that teachers could eliminate or do differently to free up valuable time for high-leverage instructional strategies (add them below as a comment!). So what could teachers do instead with the time they gain? Here are a few things:

  • Collaboratively design short assessments that would allow them to monitor student progress on key learning goals
  • Collectively examine the data that they receive from these assessments and formulate instructional modifications for learners that are still struggling
  • Participate in ongoing, long-term learning groups that help them gain new skills (or new technologies!) for addressing the needs of struggling learners

In other words, they could do things that we know to have better instructional payoff than some of what teachers are doing now. Plus, there’s the very sobering list from Mike Schmoker’s newest book, Results Now, that reminds us that teachers often don’t do things they already know have high payoffs instructionally (from a study that did thousands of classroom observations):

  • Classrooms in which there was evidence of a clear learning objective: 4 percent
  • Classrooms in which high-yield strategies were being used: 0.2 percent
  • Classrooms in which there was evidence of higher-order thinking: 3 percent
  • Classrooms in which students were either writing or using rubrics: 0 percent
  • Classrooms in which fewer than one-half of students were paying attention: 85 percent
  • Classrooms in which students were using worksheets (a bad sign): 52 percent
  • Classrooms in which noninstructional activities were occurring: 35 percent

As Schmoker notes, “such statistics point to how even fairly obvious actions could have an immediate and enormous impact on students and their levels of learning” (p. 18).

The phrase “work smarter not harder” is trite and often is used in a condescending manner. Many times, unfortunately, it also happens to be true.

This post is also available at the TechLearning blog.

We need DDDM evaluation panel participants

See below – a message I sent out over a few listservs – thought I’d post it here too. Please forward on to others and consider participating yourself (if appropriate). Do some good – earn $50!

——————–

Dear colleagues,

We are looking for teachers, principals, assessment directors, and district technology coordinators who are interested in formative assessment and are willing to participate in an evaluation panel. The purpose of the panel is to review and contribute to the design of a formative assessment tool. This project is funded by the U.S. Department of Education – its purpose is to design an affordable tool that would assist teachers with data collection and analysis on teacher-created and other formative assessments to help improve student achievement.

Each session will be no more than 1 hour long and will be conducted via webcast. Educators will receive a $50 stipend for participating.

Teachers from all subject areas are welcome. Sessions will be conducted in early October and early November. Please view the schedule at the end of this message for exact dates and times.

Educators can participate from home or from school. To participate in the webcast, each participant will need a phone to call into the discussion (toll-free) and a computer to see the presentation. Most individuals should be able to easily get into the web conference, but you may wish to check these system requirements:

  • A stable 56k, cable modem, ISDN, DSL, or better Internet connection
  • For Windows users: Internet Explorer 5.0 or newer, Netscape 6.0 or newer, Mozilla Firefox 1.0 or newer (JavaScript and Java enabled)
  • For Mac OS X users: Safari 1.3 or newer, Firefox 1.5 or newer on PowerPC G3/G4/G5, Firefox 1.5.0.2 or newer on Intel (JavaScript and Java enabled)

If you are interested in participating, please reply to this e-mail as soon as possible by clicking the reply button and typing in your answer to each of the questions below – please address your e-mail to . A confirmation message with the toll-free phone number and the web link you need to access the webcast will be sent to you.

  • Your name?
  • Your position? (teacher, principal, assessment director, district technology coordinator)
  • If a teacher or principal, are you in an elementary, middle or high school?
  • If you are a teacher, what subject do you teach?
  • In what state are you located?
  • Session date/time that is your 1st preference? (see below)
  • Session date/time that is your 2nd preference? (see below)

Thank you in advance for your assistance with this important project that aims to create helpful tools to meet educators’ data-driven accountability needs. Please share this message freely with other educators.

Dr. Scott McLeod, Assistant Professor
University of Minnesota

———————-

Session Dates / Times (all times Eastern)

Elementary School Teachers:

Oct 16 – 7:30 to 8:30 PM EST 

Oct 17 – 8:30 to 9:30 AM EST 

Oct 23 – 7:30 to 8:30 AM EST

Oct 25 – 9:30 to 10:30 AM EST

Nov 6 – 7:30 to 8:30 AM EST

Nov 8 – 9:30 to 10:30 AM EST

Nov 9 – 8:30 to 9:30 AM EST

Nov 10 – 10:30 to 11:30 AM EST

Nov 13 – 7:30 – 8:30 PM EST

Middle School/ Jr. High Teachers:

Oct 16 – 8:30 to 9:30 AM EST

Oct 16 – 7:30 to 8:30 PM EST 

Oct 17 – 10:30 to 11:30 AM EST

Oct 23 – 9:30 to 10:30 AM EST

Nov 8 – 7:30 to 8:30 AM EST

Nov 9 – 10:30 to 11:30 AM EST

Nov 10 – 8:30 to 9:30 AM EST

Nov 13 – 7:30 – 8:30 PM EST

High School Teachers:

Oct 16 – 3:30 to 4:30 PM EST

Oct 17 – 6:30 to 7:30 PM EST

Oct 23 – 8:30 – 9:30 PM EST

Nov 13 – 3:30 to 4:30 PM EST

Nov 14 – 5:30 to 6:30 PM EST

Nov 15 – 4:30 to 5:30 PM EST

Nov 16 – 8:30 to 9:30 PM EST

Principals:

Oct 16 – 1:00 to 2:00 PM EST

Oct 24 – 6:30 – 7:30 PM EST

Nov 13 – 10:00 to 11:00 AM EST

Nov 14 – 11:00 AM to 12:00 PM EST

Nov 14 – 7:30 to 8:30 PM EST

Nov 15 – 12:00 to 1:00 PM EST

Nov 16 – 1:00 to 2:00 PM EST

Assessment Directors:

Oct 17 – 8:30 – 9:30 PM EST

Oct 24 – 2:00 to 3:00 PM EST

Nov 6 – 1:00 to 2:00 PM EST

Nov 7 – 4:00 to 5:00 PM EST

Nov 8 – 3:00 to 4:00 PM EST

Nov 9 – 2:00 to 3:00 PM EST

Nov 15 – 7:30 to 8:30 PM EST

Technology Coordinators:

Oct 17 – 8:30 – 9:30 PM EST

Oct 25 – 1:00 to 2:00 PM EST

Nov 6 – 3:00 to 4:00 PM EST

Nov 7 – 2:00 to 3:00 PM EST

Nov 8 – 1:00 to 2:00 PM EST

Nov 9 – 4:00 to 5:00 PM EST

Nov 15 – 7:30 to 8:30 PM EST

Note: If none of the above dates/times work for you but you’re interested in participating, please e-mail with preferred dates and times. We may be able to add more sessions.

DDDM and Moneyball

I’ve been wanting to write this piece for a long time, but never figured out the right outlet.  This blog, however, is a great space for me to try it out (ah, the beauty of blogging!).  Plus, I think I did reasonably well with my Freakonomics analysis on Monday, so I figure it’s safe to work my thoughts through another popular non-fiction book.

Moneyball, by Michael Lewis, is a book about baseball.  So, I don’t know how many readers of this blog will have read the book.  But, for me, the book is about so much more than baseball and the lessons of the book are incredibly applicable to education.  Lewis initially set out to write a book about how the Oakland A’s, a major league baseball team on the low end of the financial resources continuum, managed to compete at such a high level seemingly every year.  There are tremendous disparities in spending across Major League Baseball, and the game has been plagued by the perception that the teams that spend the most money win the most.  The New York Yankees (my favorite team!) were the face of this perception.  The A’s seemed to defy this perception.

What Lewis ultimately discovered was that the A’s, led by their unique general manager Billy Beane, had adopted an organizational commitment to scouting and assessing players using statistical analysis of the loads of data generated by the game of baseball.  These forms of analysis, labeled generally as sabermetrics, had been around for many years, but they were largely written off as the province of geeks and statheads who just happened to like baseball.  Beane, however, came to believe that a sabermetric approach to scouting and valuing players would allow them to make the most cost-effective decisions possible. They were able to figure out which individual statistics were the greatest predictors of team success.  Then they sought players who thrived in those key areas despite being deemed by others as flawed in other areas of the game thereby devaluing their salaries.  So, for example, sabermetricians were able to demonstrate that on base percentage (OBP – the likelihood that a player will get on base in any given plate appearance) was the single greatest predictor of runs scored.  That seems logical since a player must get on base to score a run.  But, traditionally, players were valued based more on their batting average (the likelihood that a player will get a hit in any given plate appearance) than OBP.  Therefore, there were some players who didn’t have great batting averages (and as a result didn’t earn very high salaries), but had relatively higher on base percentages (mostly because they earned lots of base-on-balls – aka walks).  Those players became Oakland A’s.

Most baseball purists and old school baseball people fervently opposed this sabermetric orientation.  They argued that you couldn’t judge a player by crunching numbers.  You had to watch the players play, get to know them as people, etc.; in other words, make value determinations by scouting the old fashioned way.  The numbers were cold and unreliable, they’d say.

Is this starting to sound familiar?  Purists, traditionalists arguing that we should not rely on numerical data to make decisions?  Numerical data are cold and unreliable, and they can’t tell you what you need to know about people?  These are the same arguments you hear from those opposed to what has been labeled "data-driven decision-making" (DDDM) in education. 

I could stop here and argue that the Oakland A’s commitment to sabermetrics and cost-effective decision-making has been highly successful (just look at how well they’re doing in this year’s playoffs!), so everyone should buy in to DDDM in education.  But, that’s not the real point I want to make.

For me, there is another perception problem here.  The popular sports media has, for the most part, portrayed this so-called Moneyball philosophy inaccurately.  The popular sports media would have us believe that sabermetric analysis is an opposing paradigm to traditional baseball scouting methods.  But, the fact is that sabermetric analysis has been used by the A’s (and now many other teams as well) as a complement to more traditional methods of scouting and player valuation.  It is not as if the A’s have fired all of their scouts and hired all statisticians; their scouting department includes a few number crunchers in addition to all of the scouts who do what they’ve always done.

Similarly, in education, "data-driven decision making" is the label given to the movement to making decisions based on the scores of numerical data that are now available to educators as technological means (computers, databases, etc.) have intersected with a climate of standards and assessment.  But, to suggest that DDDM is a new movement or idea in education implies that before now, decisions were made in a vacuum; decisions were made in the absence of data.  That’s not the case, though.  Decisions, particularly those about individual students, were made based on professional judgments (teacher perceptions, observations, etc.).   Like sabermetrics in baseball, (statistically oriented) DDDM is a complementary approach to professional judgment in education.  They are epistemologically different approaches, but they are not mutually exclusive. 

Finally, the Oakland A’s needed to add sabermetric analysis to their organization because they were playing on an uneven playing field with respect to financial resources.  As a result, they have been able to compete successfully against the big spenders.  Education is a notoriously uneven playing field with respect to financial resources.  I hope schools and districts struggling with relatively low per-pupil expenditures see DDDM as a way to make more cost-effective decisions.