by jon | Oct 3, 2006 | Assessment, Guest Bloggers, Quotes, Tech Integration |
If you haven’t read the popular non-fiction book Freakonomics, I highly recommend it. Or, if it’s more your speed, you can visit the website associated with the book. The authors of the book, Steven D. Levitt and Stephen J. Dubner, even have their own blog!
Levitt and Dubner are running a series of articles in the New York Times Magazine that read much like the chapters in their book. The most recent article (free registration required, sorry) ran last week and sparked a number of thoughts about the issues around which this blog revolves. So, I thought I’d use this post to share those thoughts.
The article is about one doctor’s efforts to improve hand-hygiene compliance at Cedars-Sinai Medical Center. Dr. Leon Bender, the hospital’s Chief of Staff, along with other hospital staff, devised a number of incentive schemes to get doctors to be more diligent about washing their hands. They tried the usual campaign of memos, e-mails, etc. They formed a "Hand Hygiene Safety Posse" that walked around the wards handing out Starbucks gift cards to doctors caught in the act of washing their hands. These efforts increased compliance to about 80%, but they needed 90% compliance to satisfy their accrediting agency (really? only 90% compliance is sufficient?).
So, one day, during a meeting of Dr. Bender’s Chief of Staff Advisory Committee, the hospital’s epidemiologist took a culture of the hands of the committee members. The members pressed their palms into agar plates, and the plates were cultured and photographed. When the results came back, the images were apparently "…disgusting and striking, with gobs of colonies of bacteria.” The hospital administration decided to use the power of those images by making one particularly disgusting image the screen saver on all hospital computers. Suddenly, hand-hygiene compliance shot up to nearly 100%.
Dr. Bender’s explanation is what struck me the most. He said, "With people who have been in practice 25 or 30 or 40 years, it’s hard to change their behavior. But when you present them with good data, they change their behavior very rapidly.”
There are obvious implications of that quote for DDDM, but I’ll pass on that discussion until Thursday’s post. For today, I’m more interested in the implications for school leaders trying to facilitate technology integration (a term of art, BTW, that will be the subject of tomorrow’s post). One of the greatest challenges facing technologically-inclined school leaders is getting veteran teachers (even those who’ve been teaching the same way for 10 or so years) to change their behaviors. After Dr. Bender’s explanation, the NYT article goes on to say that "some forms of data, of course, are more compelling than others, and in this case an image was worth 1,000 statistical tables."
So, the question for me is, "what kinds of ‘data’ (broadly conceived) can we present to teachers that would compel them to change their behaviors and to integrate technology more into the teaching and learning process?" I like to show my students a PowerPoint slide that contains two photos side-by-side. One image is of a classroom from the 1880’s; the other is of a classroom from the 1980’s.
(photos from Raymond Bial’s One Room School and PBS’s evolving classroom series)
Not surprisingly, the classrooms look nearly identical. That photographic juxtaposition is usually pretty compelling. I also think David Warlick’s letter from principals is a compelling artifact.
What other data can you think of? Are there other "frightening" images or artifacts we can use to "scare" teachers (and administrators, for that matter) into changing their behaviors?
by Scott McLeod | Sep 28, 2006 | Assessment, Learning and Teaching, Research and Evaluation, Tech Integration |
Does technology positively impact student learning? Although David Warlick has noted that the question may not really matter that much, educators, citizens, and policymakers still are interested in this issue as they seek to justify current and expanded technology funding in schools.
The Metiri Group has just issued a report commissioned by Cisco Systems that finds that technology can have significant impacts on student learning outcomes when implemented with fidelity. The report, Technology in Schools: What the Research Says, notes that school organizations must pay “serious attention . . . to leadership development, professional development for teachers, school culture, curricular redesign, and teacher preparation” in order to see the learning benefits of their technology investments. The results are no big surprise for technology advocates, but it is nice to see additional research support for our claims.
Guess what? All of these issues (except teacher preparation) are under the control of school leaders. Now, what are we going to do to help administrators ramp up their technology leadership skills?
You can download the full report or read eSchoolNews’ article.
by Scott McLeod | Sep 21, 2006 | Assessment, Guest Bloggers, Research and Evaluation |
Since David posted about one of his data-driven decision-making (DDDM) projects yesterday, I thought I’d chime in with some info about a massive DDDM initiative underway here in Minnesota…
The Minnesota Statewide DDDM Readiness Study is an attempt to answer two basic questions:
- Are Minnesota teachers and administrators ready to engage in data-driven education?
- What needs do Minnesota educators have as they work to integrate data into their daily practice?
To try and answer these questions, paper surveys were sent out to every public school principal, superintendent, and technology coordinator in the state of Minnesota, including charter schools. In addition, surveys were sent out to five randomly-selected teachers in every public school building. Participants had the option to complete the survey online instead of on paper. A total of 13,850 surveys went out for this initiative; we’ve heard back from over 4,200 educators across the state. The next phase of the project hopefully will be to complete some follow-up interviews.
We’re currently doing the first round of data analysis. If you’re interested in the survey instruments themselves, they’re available here:
We believe this is the first statewide DDDM needs assessment in the country and we are delighted that the Minnesota Department of Education saw fit to fund this project. If your state has interest in using the surveys we created, we have the ability to host those online for you. The general URL for the project is www.minnesotadatasurveys.org.
by david_quinn | Sep 20, 2006 | Assessment, Guest Bloggers, Leadership and Vision, Professional Development, Research and Evaluation |
I have learned much about data-driven decision making (DDDM) from Dr. Scott McLeod. He is an acknowledged authority on DDDM and is especially knowledgeable about frequent formative assessment. My own interest in DDDM is focused on how individuals can use data to lead change in schools and create cultures that are more data-driven. I wrote a monograph for NASSP in 1999 entitled "Using Data for School Improvement". At the time I could see where the winds were blowing in the US in regards to school accountability. Looking back, I don’t think I could have predicted how far along that continuum we have traveled.
In principle, most all of us believe in the ideals of DDDM which Scott has discussed extensively on this blog. The problem is that for many policy makers and even educators, the only "data" that counts in DDDM are student test scores. While I agree that academic success should be the primary focus, this perspective dismisses so much of what teaching and learning is really about. My first grade daughter is an incredible artist (yes I am biased) and has been inspired by a wonderful public school art teacher for the last two years, but I worry that talent may not be nurtured in future grades because schools are divesting from art, music and other "non-academic" subjects to devote more resources for reading and math.
To investigate my interest in the relationships among DDDM, leadership, change, and school culture, I spent part of last year creating a survey instrument to assess teacher perceptions. I have used the survey "Data-Driven Decision Making in Schools" in ten schools so far and I have begun factor analysis procedures. One of the high school principals whose school completed the survey indicated that she has used the results to create her own professional development plan for personal growth. She is building on her strengths as a leader of DDDM and is formulating action plans to address areas of concern. I hope to conduct in-depth research and use this instrument to help school leaders understand teachers’ beliefs and learn what their teachers know or do not know about DDDM. The initial data that I have analyzed does not paint a positive picture. Not only are teachers lacking the knowledge about DDDM and concepts like frequent formative assessment. The data also indicate that teachers and even principals in many schools do not have access to the data that they need to make informed decisions about instructional practice. These are organizational and structural barriers that have to be addressed.
While educators sometimes complain about the application of business related research and theory to educational organizations, I have found much in this literature that has informed my work in DDDM. I am especially impressed with the Balanced Scorecard concept by Kaplan and Norton and the practical application that it holds for education. I am about to submit a manuscript for publication with one of my doctoral students, examining the relationship between DDDM and the Balanced Scorecard. Speaking of manuscripts, I have to take my leave to edit one now. DMQ
by Scott McLeod | Sep 13, 2006 | Assessment, Pre-Service Preparation, Professional Development, Tech Tools |
Educators today are expected to integrate the collection and analysis of student learning data into their ongoing instructional and organizational practice. Yet if you walked into almost any school in this country and asked how many teachers knew how to make a basic graph using spreadsheet software, you’d be lucky if a fourth of them raised their hands (and don’t even bother asking the administrators!). So what can we do to help educators learn some basic technology skills that can help them track and monitor student learning progress over time?
CASTLE’s School Data Tutorials web site was created to assist school districts and educational leadership preparation programs with the task of training K-12 educators to work with raw student and school data. The web site contains over 100 short Flash screencasts that walk educators through some Excel skills that are extremely helpful when working with classroom-, building-, or district-level data. Popular tutorials that are immediately useful to teachers and principals include
Most of the screencasts are only a minute or two long and can be watched a thousand times if necessary. This is extremely useful for educators who are not that tech-savvy and may not be able to keep up with the group during face-to-face training. The online availability also allows educators to return to the site later to review something if they forget what they learned previously.
We have been getting tremendously positive feedback about this resource. Educators who previously saw no intersection of spreadsheets with their professional lives are contacting us, indignant that no one has ever showed them these skills previously. Comments like "I could have saved myself a lot of time" or "Now I have a way to do this easily" are extremely common. One of the reasons I think the tutorials are so well-received is because they’re grounded in the important, real-life, data-driven decision-making needs of teachers and administrators (as opposed to the generic, context-less Excel training that occurs in many school systems).
This summer we added a section to the site that walks educators through the steps of creating their own data collection templates. This section, which is not quite as polished as the rest of the site (e.g., I think you can hear my in-laws’ computer’s aquarium screensaver in the background!), contains examples for principals, teachers, and grade-level teams. In the coming months, we will upload additional examples of templates that are being used in schools across the country.
The tutorials currently are being used in schools, districts, and university preparation programs across the nation. I encourage you to check out the site (maybe you’ll learn something helpful!) and to spread the word about this resource. We created it to help build some capacity in school systems. Right now data analysis expertise resides primarily in central office research and assessment departments – we need more knowledge in our buildings and classrooms.
More information is available at the web site if you’d like to know more about this resource or our critical partners. Any feedback or suggestions that you have are extremely welcome – simply comment on this post!
by Scott McLeod | Sep 12, 2006 | Assessment, Management and Operations, Tech Tools |
In addition to the white paper that I wrote for Microsoft that summarized essential data-driven decision-making concepts for teachers and principals, I also wrote a second white paper that summarized the various technology tools that school districts are using to manage summative and formative data.
Technology Tools for Data-Driven Teachers is only seven pages long and is a brief overview of data management and analysis (DMA) systems (i.e., data warehouses) and instructional management and assessment (IMA) systems. DMA systems are primarily used for management and analysis of summative data, such as those from yearly state assessments, and typically interface with districts’ student information systems, electronic gradebooks, and other district databases. IMA systems typically are not integrated with DMA systems and are used to separately store and analyze data from ongoing progess monitoring efforts. I also briefly discuss in the white paper some other options that school districts have if they can’t afford these often-expensive data systems.
Technology Tools for Data-Driven Teachers is a nice overview for educators who are unfamiliar with the technology options available for storing and analyzing student and school data. The white paper is deliberately written in non-technical language to facilitate easy understanding by administrators and teachers who probably are not technology wonks. Whether your district is reexamining existing tools for managing data or is looking for new solutions, the white paper can help give your educators get the big picture of what’s out there as they try to make sense of different vendor offerings.
Two other resources that may be helpful are my ongoing lists of DMA system vendors and IMA system vendors. School districts often don’t have the ability or inclination to do a full market scan of possible products and thus often miss vendor solutions that may better fit their needs. I don’t claim that my categorization is perfect – sometimes products blur the lines between categories. If you have a vendor to add to one of these two lists, please comment on this post!
by Scott McLeod | Sep 7, 2006 | Assessment |
I’ve been getting a lot of positive feedback on the white paper that I wrote last year for Microsoft so I thought I would share it as a resource here. I wrote Data-Driven Teachers with the intent of trying to concisely summarize the great work being done on data-driven school reform by folks such as Mike Schmoker, Rick DuFour, Victoria Bernhardt, Douglas Reeves, Rick Stiggins, etc.
Data-Driven Teachers is only ten pages long and utilizes the data-driven change framework that I have used with thousands of educators from across the country. Each section of the paper describes what teachers and principals should be doing for that particular data-driven concept. The last page of the paper is a table from a draft of the Professional Development Framework for Data-Driven School Improvement that I created for the Chicago (IL) Public Schools.
Data-Driven Teachers is a nice introduction to the concepts of data-driven school reform and is a good reminder of some essential activities that often are missing in school organizations. School districts have been using it to quickly ramp up their teachers and administrators conceptually, to give their educators a common framework and language for discussion, and to remind staff of key actions that need to occur as they work to better utilize student learning data to improve instructional and organizational decision-making.
If you find the white paper useful, drop me a comment!
by Scott McLeod | Aug 31, 2006 | Assessment, Professional Development |
In a previous post, I commented on the perceptions of many K-12 educators that their school’s academic success is hostage to their student demographics.
There’s another angle to this – the belief by some (but, thankfully, not most) educators that current achievement levels are just fine. I ran across this again recently when I spoke to a school district’s entire teaching staff about utilizing formative assessment data to improve overall learning outcomes and close existing achievement gaps. Frequent formative assessment has been shown by countless research studies, and by successful data-driven schools across the country, to have powerful impacts on student achievement, particularly that of low achievers.
One of the teachers left this comment on her anonymous index card at the end of my presentation:
I’m glad that my children don’t attend a school where it’s okay for a teacher to believe that leaving 1 in 5 kids behind is just fine. The sad and scary fact is that there are entire communities that have expectations this low (or even lower) for their children.
by Scott McLeod | Aug 24, 2006 | Assessment, Law, Policy, and Ethics, Leadership and Vision, Quotes |
A recent post by Kevin Carey at The Quick and the Ed highlights one of the essential dilemmas faced by those of us who are working desperately to improve students’ academic and life success: there is a pervasive attitude in K-12 organizations that outputs are dependent on inputs. You routinely hear comments from educators such as "You can’t expect us to do any better than we already are with these kids" or "The reason that [school / district] is doing better than we are is because they serve those kids."
"We believe that we have no meaningful impact on the children that we serve. We are hostage to our demographics. Whatever comes in the door is essentially what’s going to go out at the other end." Those are chilling words to hear, both as an educational leader and as a citizen of the most affluent and powerful nation in the world.
Despite the ritualistic mantra of educators that "all children can learn," there are large numbers of teachers and principals who don’t truly believe it. If they did, they would act in ways much differently than they do now.
Luckily we have (increasingly numerous) examples of schools where this belief has been challenged at its very core, where educators have come together and said "Collectively we can make a difference!" These schools are finding ways to make it happen, despite their challenging demographics. It starts with a belief that it can be done. As Dr. Douglas Reeves notes in The Learning Leader:
Norfolk Public Schools in Virginia has the following demographic characteristics:
- 80 percent of students receive free or reduced-priced lunch
- 68 percent minority student enrollment
- 40+ languages
Between 1998 and 2005 not a single child in the school system, to the best of my knowledge, has changed his or her ethnic identity. Not a single child has won the lottery. Few if any children have adopted different languages at home. In other words, this story is not about changes in children or their families, nor is it a story about changes in demographic characteristics. This is a story about changes in teaching, leadership, and learning. While demographic characteristics remained the same, student achievement rose dramatically.
In 1998, only 11 percent of the elementary schools in Norfolk contained more than 50 percent of students who scored proficient or higher on the state’s English/Language Arts assessments. In 2004, 84 percent of the elementary schools achieved that distinction, and in 2005, 100 percent of the elementary schools in the district were fully accredited. They not only had 50 percent of students meeting or exceeding state standards in English / Language Arts, but these students also met state requirements in math, science, writing, and social studies. In 1998, none of the middle schools in this district had more than 50 percent of students meeting state English / Language Arts requirements, and six years later all the middle schools met this requirement. In addition, the district more than tripled the number of middle school students taking advanced math courses in middle school. In 1998, only one out of six high schools had more than 80 percent of students passing state English graduation requirements, and six years later every high school in the system achieved this distinction. Moreover, some high schools had more than 90 percent of students passing external exams in chemistry and biology, while the dropout rate remained an astonishingly low 0.5 percent for the district – one of the lowest high school dropout rates of any urban system in the nation. The students didn’t change. They were still ethnically, linguistically, and economically diverse. But something profound did change – the commitment of the leaders and teachers in this district to make a difference in the lives of students.
Yes, we absolutely need to be sensitive to overall contexts and larger societal issues. I, too, am concerned about the increasing gaps between the top and bottom income brackets of our society. I, too, am gravely worried that our political and educational policies may be causing increasing harm to our most disadvantaged populations. We must seek to understand the complex contexts in which our students live and we must fight and advocate and cooperate until progress is made. But we must never, ever give up, even mentally. For then the battle, and all hope, is lost.
