by Scott McLeod | Mar 2, 2022 | Equity, Law, Policy, and Ethics, Research and Evaluation |
Hechinger Report just published an article on how having teachers study student data doesn’t actually result in better student learning outcomes.
Think about that for a minute. That finding is pretty counterintuitive, right? For at least two decades now we have been asking teachers to take summative and formative data and analyze the heck out of them. We create data teams and data walls. We implement benchmarking assessments and professional learning communities (PLCs). We make graphs and charts and tables. We sort and rank students and we flag and color code their data… And yet, research study after research study confirms that all of it has no positive impact on student learning:
[Heather Hill, professor at the Harvard Graduate School of Education] “reviewed 23 student outcomes from 10 different data programs used in schools and found that the majority showed no benefits for students” . . . . Similarly, “another pair of researchers also reviewed studies on the use of data analysis in schools, much of which is produced by assessments throughout the school year, and reached the same conclusion. ‘Research does not show that using interim assessments improves student learning,’ said Susan Brookhart, professor emerita at Duquesne University and associate editor of the journal Applied Measurement in Education.”
All of that time. All of that energy. All of that effort. Most of it for nothing. NOTHING.
No wonder the long-term reviews of standards-, testing-, and data-oriented educational policy and reform efforts have concluded that they are mostly a complete waste. We’re not closing gaps with other countries on international assessments. Instead, our own country’s achievement gaps are widening. The same patterns are occurring with our own national assessments here in the United States. Similarly, our efforts to ‘toughen’ teacher evaluations also show no positive impact on students. It’s all pointless. POINTLESS.
The past two decades have been incredibly maddening and demoralizing for millions of educators and students. And for what? NOTHING.
Are school administrators even paying attention? Or are they still leaning into outdated, unproductive paradigms of school reform?
This was the line in the article that really stood out for me:
Most commonly, teachers review or re-teach the topic the way they did the first time or they give a student a worksheet for more practice drills.
In other words, in school after school, across all of these different studies, our response to students who are struggling is to… do the same thing again. Good grief.
Make school different.
—
MARCH 8 ADDENDUM
Here are some additional paragraphs from the Hill article:
Goertz and colleagues also observed that rather than dig into student misunderstandings, teachers often proposed non-mathematical reasons for students’ failure, then moved on. In other words, the teachers mostly didn’t seem to use student test-score data to deepen their understanding of how students learn, to think about what drives student misconceptions, or to modify instructional techniques.
Field notes from teacher data-team meetings suggest a heavy focus on “watch list” students—those predicted to barely pass or to fail the annual state reading assessment. Teachers reported on each student, celebrating learning gains or giving reasons for poor performance—a bad week at home, students’ failure to study, or poor test-taking skills. Occasionally, other teachers chimed in with advice about how to help a student over a reading trouble spot—for instance, helping students develop reading fluency by breaking down words or sorting words by long or short vowel sounds. But this focus on instruction proved fleeting, more about suggesting short-term tasks or activities than improving instruction as a whole.
Common goals for improving reading instruction, such as how to ask more complex questions or encourage students to use more evidence in their explanations, did not surface in these meetings. Rather, teachers focused on students’ progress or lack of it. That could result in extra attention for a watch-list student, to the individual student’s benefit, but it was unlikely to improve instruction or boost learning for the class as a whole.
I think my takeaways from all of this are that:
- As would be expected, analyzing student data alone doesn’t do much for us. We also need to have effective interventions.
- Despite our best intentions and rhetoric, the research indicates that most schools don’t actually engage in effective interventions.
So all of our data-driven, PLC, RTI, etc. work isn’t actually doing much for us, at least in terms of student learning outcomes. Learning gaps continue to persist. Teacher instruction isn’t changing. And so on…
Image credit: Wincing, Frédéric Poirot
by Scott McLeod | Sep 7, 2020 | Blogging, Research and Evaluation |
Just leaving these two tables here as I reflect on the impact of my work and where I choose to put it. The scale isn’t even close. And this doesn’t even factor in interactivity… (e.g., my 80 blog posts that have received at least 30 comments, including one that has received 618!)
Over 4.4 million page views and counting!
by Scott McLeod | May 20, 2019 | Leadership and Vision, Learning and Teaching, News and Events, Planning and Funding, Research and Evaluation |
I am pleased to announce that my book chapter in Multimedia Learning Theory: Preparing for the New Generation of Students is now available!
My chapter is titled Multimedia Learning and the Educational Leader. Here’s an excerpt from the Systemic Improvement section of the chapter:
One important role of principals and superintendents is ensuring that employee position announcements, job descriptions, hiring processes, mentoring systems, training, and evaluation criteria all enforce school organizations’ need for robust multimedia learning and teaching. Few school systems currently have powerful technology integration as a core competency for classroom teaching staff. Educational organizations’ omission of digital teaching proficiency and the ability to facilitate students’ higher-order thinking as essential, required skill sets for teachers and administrators sends clear signals to current faculty, job candidates, and educator preparation programs about institutional values. Until this changes, meaningful and authentic uses of multimedia technologies in P-12 classrooms will continue to be isolated aberrations rather than routine observances.
Over the past several decades, school systems have slowly instituted a variety of technical systems to facilitate the management of lower-level cognitive work. Student information systems, online gradebooks, electronic formative assessment tools, and data warehouses are all examples of technologies that help educators input, manage, analyze, and present student learning data (Wayman, Stringfield, & Yakimowski, 2004). Most of the data in these institutional systems focus on student demographic information, letter grades, test scores, and daily assignment tracking.
As educational organizations transition to student learning environments that place greater emphasis on higher-order thinking skills, they will need more robust technology tools that allow them to facilitate, collect, and evaluate more complex, abstract, open-ended student learning. Most of these tools do not yet exist, so it is difficult to envision at this time what they might look like. They are likely to include evolving features such as sophisticated document and portfolio management (including archiving and tagging of multimedia student and teacher work products); deep cross-artifact text, image, audio, and video analysis; infographic-like presentation of underlying patterns and meaning; and the ability to easily but selectively share through a variety of information and social media channels.
Other tools to facilitate effective learning and teaching will include system-provided technologies such as open access content repositories, streaming multimedia servers, online adaptive learning systems, and robust, social media-driven collaboration channels for students, classroom teachers, and administrators. Strategic partnerships with state and federal governments, corporations, foundations, nonprofits, and others will become more prevalent as school systems face inevitable gaps in funding and resources. Growth in these and other technology systems must be accompanied by concurrent growth in organizational thinking as well as administrative and societal permission and encouragement to utilize these tools.
Hope the book is useful to some of you. Happy reading!
Citation
McLeod, S. (2019). Multimedia learning and the educational leader. In P. M. Jenlink & B. D. Knight (Eds.), Multimedia learning theory: Preparing for the new generation of students, pp. 129-143. Lanham, MD: Rowman and Littlefield.
by Scott McLeod | Nov 30, 2018 | Higher Education, Our Changing World, Research and Evaluation |
I had the privilege of participating in a conversation today at my university about how (and whether) the digital work done by faculty should count for promotion and tenure. (I also had the opportunity to speak for a few minutes; here are my slides). Here are four thoughts that are spinning around in my brain after a couple of hours of discussion…
First, if we publish an article in a traditional journal which happens to post that article online and it then gets a few social media shares, that does not make us ‘digital scholars.’ That’s definitely a step beyond traditional analog publishing. But to be relevant to the digital, online, hyperconnected, participatory, interactive, highly-distributed information landscape in which we now live and work, we need to expect more from ourselves. Not all of the time, but sometimes. And more often.
Second, why is it that the faculty who ARE trying to be relevant in our new information landscape are the ones that always have to justify their work to those who are less responsive? Shouldn’t it be the other way around? Which side should carry the burden of persuasion regarding relevance and quality?
Third, no one owes us anything, no matter how good the work is that we do. We have to prove ourselves every day. Just because we did good work a decade ago doesn’t mean that we are doing so now. Just because we believe that our work is valuable doesn’t mean that others do – or should. As Seth Godin says, “If [our] target audience isn’t listening, it’s [our] fault, not theirs.” Make the case. Be engaged. Help others see the meaning and value in what we do. All the time. (These digital tools can help…)
Finally, we need to be less dismissive of the public and of publishing for non-academics. We ignore engagement with the public and policymakers at our peril.
With appreciation for all of the complexities behind these fairly simple assertions… let me know what you think.
by Scott McLeod | Sep 16, 2017 | Research and Evaluation, Social Media, Youth and Media |
Information literacy has been a hot topic of recent conversation. Many folks believe that web sites that traffic in false information and ‘fake news’ may have influenced the last United States presidential election. Traffic on the Snopes web site, which debunks false rumors, has never been greater. Ideological separation also is being driven by the ways that we sort ourselves in our schools, neighborhoods, friendship groups, political affiliations, and faith institutions. Already often isolated from the dissimilar-minded, we then also self-select into individualized news media and online channels that can result in walled-garden ‘echo chambers’ or ‘filter bubbles.’
To combat our growing concerns about fake news and filter bubbles, we’re going to have to take the task of information literacy more seriously. And that means rethinking some organizational and technological practices. As I noted in a previous blog post, our information landscape is changing both rapidly and drastically. Today we have a digital, online, hyperconnected, interactive, global information landscape that often is free or low-cost, fosters decentralized creation and participation and sharing, is frequently real-time, and has exponential reach. This landscape stands in sharp contrast to our older analog landscape that relied on ink on paper rather than bits in the ether, was expensive and thus primarily oriented around experts, fostered consumption and scarcity, and was fairly static and slow to change. As learning institutions bestowed with the societal charge of preparing informed citizens and knowledge workers, schools must help their students and graduates master the dominant information landscape of today and tomorrow, not just yesterday. And right now most schools are struggling…
School leaders can do several things to foster information literacy, combat fake news, and increase students’ information and technology fluency. One critical leadership behavior is helping educators understand that information literacy is everyone’s job, not just that of the librarian or media specialist. Being an informed citizen, being a critical thinker, being able to deeply and thoughtfully analyze complex texts – these have all been traditional student roles in schools but they are taking new forms in our emerging information spaces. Given the complexity of our new information landscape, we no longer can trot students down to the media center a few times a year to learn from the librarian about trusted voices, credible sources, and appropriate citation. All educators now must integrate information literacy in authentic and meaningful ways into ongoing digital and online work with students. Using our disciplinary expertise and experience, we thus can appropriately contextualize critical discernment. In other words, we must help our students dissect and understand subject-specific media such as false videos about the environment or websites dedicated to political untruths or viral myths about health care while they have us available as content area experts to help guide them.
School leaders also must recognize that in order for students to be actively engaged in – and critical consumers of – digital and online information channels, they must have access to technologies and online environments that often are heavily filtered or completely blocked. We can’t help our graduates be citizens and critical thinkers within spaces to which they don’t have access. This is particularly true if we want students to be actively involved within political, scientific, and other digital spaces rather than passive recipients. For instance, teaching online information literacy by pre-selecting a small handful of resources for students to analyze is vastly different from teaching students to navigate and make sense of our vast, complex online information commons.
School leaders also must create safe spaces for teaching and learning about controversial topics. Imagine, for instance, a high school government teacher who asked her students to follow the primary social media channels of the two primary political parties here in the United States. On the Republican side, students could follow GOP websites, Twitter feeds, and YouTube videos and subscribe to conservative blogs such as RedState, HotAir, Instapundit, and Michelle Malkin. On the Democratic side, students also could follow relevant websites, Twitter feeds, and YouTube channels, along with liberal blogs such as Daily Kos, The Huffington Post, Democratic Underground, and ThinkProgress. Sprinkle in a few other sites such as The Hill, Politico, FiveThirtyEight, Fox News, and CNN and we can see how real-time social media could be an incredibly powerful lens through which to view, discuss, and understand government in action, not just as abstract concepts from a dry textbook. I’m not sure how many teachers would be willing to try this, however, given schools’ traditional aversion to anything controversial. Principals and school systems must be willing to buffer a few anxieties in order to enable these kinds of meaningful learning experiences.
Schools also have to stop treating students as ‘digital natives’ who already are knowledgeable about and proficient with technology. Youth fluency with social and gaming technologies may imply certain levels of technology comfort but does not mean that students have the ability to use digital tools in academic- and work-productive ways. Not only is the ‘digital natives’ concept disproven by research, it also seems to grant us permission as educators to avoid the difficult challenge of fostering technology- and information-fluent students because we supposedly have little to teach them. Schools’ reluctance to own this challenge – perhaps because of our educators’ own lack of technology fluency – results in findings like the recent study from Stanford University that showed that students’ current information literacy skills are abysmal.
Finally, school leaders should recognize that those teachers who enable youth to actively interact and create online also are creating opportunities for students to learn essential lessons about responsible participation, sharing, contribution, etiquette, and digital citizenship as natural extensions of their classwork. This approach is far more meaningful and impactful than a few isolated media literacy sessions or digital citizenship lectures. We say that we want engaged citizens and critical thinkers. So let’s do a better job of preparing our students to be thoughtful consumers and active contributors within our new technology-suffused information spaces.
What is your school doing to help students with fake news and filter bubbles?
[cross-posted at Front and Central]
Image credit: Fake news figure, Stuart Rankin
by Scott McLeod | Oct 27, 2016 | Blogging, Higher Education, Research and Evaluation |
Back in 2008 I raised the questions:
Why would anyone who wishes to actually reach educators and hopefully influence change in schools not be blogging?
Also… why haven’t more faculty caught on to this?
Eight years later, I thought that I would share a couple of recent tables that I made in order to illustrate this point further. The first table is the number of academic citations that I have received on the top 20 things that I have written or created (according to Google Scholar). My citation numbers are decent if not spectacular; they’ve been enough to get me tenure at several of our nation’s top research institutions.
[click here for a larger image]
The second table shows the number of page views and comments that I have received on my top 20 blog posts.
[click here for a larger image]
No comparison in terms of reach, visibility, interaction, and (hopefully) impact. The percentage of university faculty members who are blogging – although better than it was 8 years ago – is still incredibly low. We pay the price in terms of public and policymaker awareness of and attention to our work.
by Scott McLeod | Dec 16, 2015 | Law, Policy, and Ethics, Research and Evaluation |
My local high school recently was named the top high school in Iowa by Niche.com, a school and college ranking site.
Today the Iowa Department of Education issued its first-ever school report cards. Ames High School didn’t do as well this time, only managing an overall ranking of Commendable, which is the third-highest report card category. Here are the number of Iowa schools in each of the six possible report card categories:
For this first year, the Department of Education distributed schools along a normal curve. In future years, the point boundaries for the school report categories will be locked into place and schools will be able to move in and out of the categories. In other words, down the road it is possible that some report card categories may have few or no schools in them.
I downloaded the Department’s school report card data and combined them with its free lunch data. Free or reduced-price lunch percentages often are used as indicators of school poverty. Here is what the free lunch percentage distributions look like for each report card category:
Zero of the 34 Priority schools have less than 33% free lunch eligibility and 30 of the 34 (88%) have more than half of their students who are eligible. In contrast, 27 of the 35 Exceptional schools (77%) have less than 33% free lunch eligibility and only 3 of the 35 (9%) have more than half of their students who are eligible. Here are the median and average free lunch eligibility percentages for each report card category:
Here is the box plot for each school report card category:
Here’s a reminder on how to interpret a box plot:
Iowa’s school report card results mirror those of other states, which typically show strong negative relationships between overall school report card scores and school poverty levels. So we now have an Iowa school report card system that confirms what we already knew from the peer-reviewed research and from other locations, which is that schools with higher poverty levels tend to do less well on indicators of school success. Whether we will actually do anything about it remains an open question…
Please check over my data and see if I made any mistakes. Also see my copyright policy and feel free to use these data and images as you wish for your own projects!
by Scott McLeod | May 18, 2015 | Leadership and Vision, Learning and Teaching, Research and Evaluation, Tech Integration |
Murphy & Beland’s recent study is making the rounds online, particularly among those who are eager to find reasons to ban learning technologies in classrooms. The economists found that banning mobile phones helped improve student achievement on standardized test scores, with the biggest gains seen by low-achieving and at-risk students. Here are my thoughts on this…
The outcome measure is standardized test score improvement. Is that all you care about or do you have a bigger, more complex vision for student learning? For instance, creativity, collaboration, critical thinking, and problem-solving are difficult to assess with a standardized test. Most schools I know didn’t adopt their learning technology initiatives for the sole purpose of test score improvement. (if they did, how sad is that?)
The accepted dichotomy in this study and the media seems to be 1) doing low-level knowledge work while smartphones are banned, or 2) doing low-level knowledge work while smartphones are present (and, presumably, distracting). Neither of these two options addresses the fact that decontextualized, low-level work isn’t very interesting or engaging to many (most?) students, particularly those who already find that traditional schooling doesn’t meet their needs very well. So, faced with the opportunity to do something else, many students do. Youth today aren’t any different than when we were young and adults made snarky, woeful comments about us. They just have different opportunities and resources. How many times were you bored in high school? Lots, so admit that if you’d had access to a smartphone or your friends on Facebook back then, you would have turned that way too. I know that I sure would have. Let’s stop blaming students and/or demonizing technology as an evil succubus and address the real problem, which is disengaging learning environments. The solution to that problem is not to try and force students to pay attention to and comply with our boring lessons. That’s not teaching students ‘grit.’ That’s an indictment of our failure to differently imagine learning and teaching.
How about a third option, that of doing higher-level learning and USING the smartphones to help with that? That sounds pretty good to me. Why isn’t this ever brought up as an option to be considered?
Image credit: Cell phone prohibited, SmartSign
by Scott McLeod | Jun 3, 2013 | Higher Education, Leadership and Vision, Research and Evaluation |
Remember last September when I made available for free my most recent book chapter, Supporting Effective Technology Integration and Implementation?
Well, that chapter’s still available to you for free, but I’m pleased to announce that the rest of the book is now available as well. Featuring a diverse set of authors, perspectives, and topics, there’s something for everybody in Principal 2.0: Technology and Educational Leadership.
Here are the chapter titles and authors (including all 4 CASTLE directors!):
- Foreword: Ready Set Go! with Educational Technology, Governor Beverly Perdue.
- Introduction: Making the Case for Principal 2.0, Jennifer Friend and Matthew Militello.
- Augmenting Educational Realities, John Militello.
- The Role of For-Profit Firms in the Educational Ecosystem, Michael J. Schmedlen.
- Generation X Meets Generation Y: Reflections on Technology and Schooling, Jennifer Friend and Alexander David Friend.
- Zen and the Art of Technology in Schools: Multigenerational Perspectives, Matthew Militello, Ronald Militello, Dominic Militello, Luke Militello, and Gabriel Militello.
- Digital Storytelling for Critical Reflection: An Educational Leadership Story, Francisco Guajardo, Miguel A. Guajardo, John A. Oliver, Mónica M. Valadez, and Mark Cantu.
- Engaging Youth Voice: Collaborative Reflection to Inform School Relationships, Processes, and Practices, Christopher Janson, Sejal Parikh, Jacqueline Jones, Terrinikka Ransome, and Levertice Moses.
- There’s an App for That: 50 Ways To Use Your iPad, April Adams and Jennifer Friend.
- Student-Owned Mobile Technology Use in the Classroom: An Innovation Whose Time Has Come, Tricia J. Stewart and Shawndra T. Johnson.
- Affective Learning Through Social Media Engagement, David Ta-Pryor and Jonathan T. Ta-Pryor.
- The Central Texas Community Learning Exchange Digi-Book: Fostering School and Community Engagement Through the Creation of a Digital Book, Lee Francis, IV, Mónica M. Valadez, John A. Oliver, and Miguel A. Guajardo.
- Leaders Online: Enhancing Communication with Facebook and Twitter, John B. Nash and Dan Cox.
- Balancing Effective Technology Leadership with Legal Compliance: Legal Considerations for Principal 2.0, Justin Bathon and Kevin P. Brady.
- Connected Principals: In Pursuit of Social Capital via Social Media, Candice Barkley and Jonathan D. Becker.
- School Leaders’ Perceptions of the Technology Standards, Matthew Militello and Alpay Ersozlu.
- Supporting Effective Technology Integration and Implementation, Scott McLeod and Jayson W. Richardson.
A big thanks to Drs. Matt Militello and Jennifer Friend for inviting me to be a part of this publication. Happy reading!
by Scott McLeod | Mar 15, 2013 | Law, Policy, and Ethics, Research and Evaluation |
One of the Iowa Department of Education’s favorite phrases is ‘evidence-based,’ as in “schools and educators should be adopting ‘evidence-based’ practices.” That makes sense on its surface, right? So when the DE issued a press release today touting two Iowa districts that had adopted the National Institute for Excellence in Teaching’s (NIET) Teacher Advancement Program (TAP) – which focuses on teacher leadership and performance-based teacher compensation – I started digging around in Google Scholar for research about the program. I don’t live in the world of teacher leadership/compensation and thus don’t know much about TAP other than that it appears to support the DE’s policy proposals this legislative session (and that Dr. Brad Buck, an amazing school leader that I greatly respect, is the superintendent of one of the two districts). For all I know, TAP could be amazing or it could be puffery…
What did I find?
And, of course, on its web site I found research from NIET itself claiming that TAP is awesome. Indeed, there’s a plethora of publications on TAP’s web site from NIET; agencies such as the Milken Family Foundation, the Joyce Foundation, or the Gates Foundation that have funded TAP; and ideologically-oriented educational organizations. But despite TAP’s claims of ‘a decade-long track record of growth and success in raising student achievement in high-need schools‘ (page 19), the evidence from independent researchers is decidedly more mixed and leans much more pessimistic when it comes to student learning outcomes.
So is TAP ‘evidence-based’ or puffery? Given the greater preponderance of negative findings by outside, independent scholars, it appears to be extremely arguable. Given that uncertainty and its own call for ‘evidence-based’ practices, should DE be basing its 2013 education reform package on the TAP model? (and should it be touting the model via press releases to the Iowa public and media, which know even less about all of this than I do?)
We already know that DE is willing to ignore decades worth of peer-reviewed research when it conflicts with its policy advocacy (see, e.g., recent incidents regarding 3rd grade retention and cutting elementary school recess). I don’t know if TAP falls into this category of willful blindness or not. But I am wondering if the studies listed above were ever presented to those in decision-making positions so that they could make a truly informed decision.
I hope that the initiatives go well for the two Iowa districts that are trying TAP. Their educators are going to invest a great deal of time, energy, and money in the model. Hopefully they will see the results for which they are striving. Until I see further independent research supporting TAP (send it if you’ve got it; maybe I missed some!), right now I’m less sanguine about whether we should be basing tens of millions of dollars statewide to implement the model here in Iowa.
Got any thoughts on this?
[Note that there’s also a bigger question here that’s always worth considering: What should we do if/when our own governmental agencies fail to apply the same standards to themselves that they wish to apply to us?]
Image credit: What’s your superpower?
