Archive | Research and Evaluation RSS feed for this section

Reflecting on my work: Google Scholar v. Google Analytics

Just leaving these two tables here as I reflect on the impact of my work and where I choose to put it. The scale isn’t even close. And this doesn’t even factor in interactivity… (e.g., my 80 blog posts that have received at least 30 comments, including one that has received 618!)

Over 4.4 million page views and counting!

2020 09 07 Google Scholar

2020 09 07 Google Analytics

New book chapter in Multimedia Learning Theory!

Multimedia Learning Theory: Preparing for the New Generation of Students (edited book)I am pleased to announce that my book chapter in Multimedia Learning Theory: Preparing for the New Generation of Students is now available!

My chapter is titled Multimedia Learning and the Educational Leader. Here’s an excerpt from the Systemic Improvement section of the chapter:

One important role of principals and superintendents is ensuring that employee position announcements, job descriptions, hiring processes, mentoring systems, training, and evaluation criteria all enforce school organizations’ need for robust multimedia learning and teaching. Few school systems currently have powerful technology integration as a core competency for classroom teaching staff. Educational organizations’ omission of digital teaching proficiency and the ability to facilitate students’ higher-order thinking as essential, required skill sets for teachers and administrators sends clear signals to current faculty, job candidates, and educator preparation programs about institutional values. Until this changes, meaningful and authentic uses of multimedia technologies in P-12 classrooms will continue to be isolated aberrations rather than routine observances.

Over the past several decades, school systems have slowly instituted a variety of technical systems to facilitate the management of lower-level cognitive work. Student information systems, online gradebooks, electronic formative assessment tools, and data warehouses are all examples of technologies that help educators input, manage, analyze, and present student learning data (Wayman, Stringfield, & Yakimowski, 2004). Most of the data in these institutional systems focus on student demographic information, letter grades, test scores, and daily assignment tracking.

As educational organizations transition to student learning environments that place greater emphasis on higher-order thinking skills, they will need more robust technology tools that allow them to facilitate, collect, and evaluate more complex, abstract, open-ended student learning. Most of these tools do not yet exist, so it is difficult to envision at this time what they might look like. They are likely to include evolving features such as sophisticated document and portfolio management (including archiving and tagging of multimedia student and teacher work products); deep cross-artifact text, image, audio, and video analysis; infographic-like presentation of underlying patterns and meaning; and the ability to easily but selectively share through a variety of information and social media channels.

Other tools to facilitate effective learning and teaching will include system-provided technologies such as open access content repositories, streaming multimedia servers, online adaptive learning systems, and robust, social media-driven collaboration channels for students, classroom teachers, and administrators. Strategic partnerships with state and federal governments, corporations, foundations, nonprofits, and others will become more prevalent as school systems face inevitable gaps in funding and resources. Growth in these and other technology systems must be accompanied by concurrent growth in organizational thinking as well as administrative and societal permission and encouragement to utilize these tools.

Hope the book is useful to some of you. Happy reading!

Citation

McLeod, S. (2019). Multimedia learning and the educational leader. In P. M. Jenlink & B. D. Knight (Eds.), Multimedia learning theory: Preparing for the new generation of students, pp. 129-143. Lanham, MD: Rowman and Littlefield.

Four quick thoughts for higher education faculty

Tivoli CU DenverI had the privilege of participating in a conversation today at my university about how (and whether) the digital work done by faculty should count for promotion and tenure. (I also had the opportunity to speak for a few minutes; here are my slides). Here are four thoughts that are spinning around in my brain after a couple of hours of discussion…

First, if we publish an article in a traditional journal which happens to post that article online and it then gets a few social media shares, that does not make us ‘digital scholars.’ That’s definitely a step beyond traditional analog publishing. But to be relevant to the digital, online, hyperconnected, participatory, interactive, highly-distributed information landscape in which we now live and work, we need to expect more from ourselves. Not all of the time, but sometimes. And more often.

Second, why is it that the faculty who ARE trying to be relevant in our new information landscape are the ones that always have to justify their work to those who are less responsive? Shouldn’t it be the other way around? Which side should carry the burden of persuasion regarding relevance and quality?

Third, no one owes us anything, no matter how good the work is that we do. We have to prove ourselves every day. Just because we did good work a decade ago doesn’t mean that we are doing so now. Just because we believe that our work is valuable doesn’t mean that others do – or should. As Seth Godin says, “If [our] target audience isn’t listening, it’s [our] fault, not theirs.” Make the case. Be engaged. Help others see the meaning and value in what we do. All the time. (These digital tools can help…)

Finally, we need to be less dismissive of the public and of publishing for non-academics. We ignore engagement with the public and policymakers at our peril.

With appreciation for all of the complexities behind these fairly simple assertions… let me know what you think.

How school leaders can combat ‘filter bubbles’ and ‘fake news’

Fake news

Information literacy has been a hot topic of recent conversation. Many folks believe that web sites that traffic in false information and ‘fake news’ may have influenced the last United States presidential election. Traffic on the Snopes web site, which debunks false rumors, has never been greater. Ideological separation also is being driven by the ways that we sort ourselves in our schools, neighborhoods, friendship groups, political affiliations, and faith institutions. Already often isolated from the dissimilar-minded, we then also self-select into individualized news media and online channels that can result in walled-garden ‘echo chambers’ or ‘filter bubbles.

To combat our growing concerns about fake news and filter bubbles, we’re going to have to take the task of information literacy more seriously. And that means rethinking some organizational and technological practices. As I noted in a previous blog post, our information landscape is changing both rapidly and drastically. Today we have a digital, online, hyperconnected, interactive, global information landscape that often is free or low-cost, fosters decentralized creation and participation and sharing, is frequently real-time, and has exponential reach. This landscape stands in sharp contrast to our older analog landscape that relied on ink on paper rather than bits in the ether, was expensive and thus primarily oriented around experts, fostered consumption and scarcity, and was fairly static and slow to change. As learning institutions bestowed with the societal charge of preparing informed citizens and knowledge workers, schools must help their students and graduates master the dominant information landscape of today and tomorrow, not just yesterday. And right now most schools are struggling…

School leaders can do several things to foster information literacy, combat fake news, and increase students’ information and technology fluency. One critical leadership behavior is helping educators understand that information literacy is everyone’s job, not just that of the librarian or media specialist. Being an informed citizen, being a critical thinker, being able to deeply and thoughtfully analyze complex texts – these have all been traditional student roles in schools but they are taking new forms in our emerging information spaces. Given the complexity of our new information landscape, we no longer can trot students down to the media center a few times a year to learn from the librarian about trusted voices, credible sources, and appropriate citation. All educators now must integrate information literacy in authentic and meaningful ways into ongoing digital and online work with students. Using our disciplinary expertise and experience, we thus can appropriately contextualize critical discernment. In other words, we must help our students dissect and understand subject-specific media such as false videos about the environment or websites dedicated to political untruths or viral myths about health care while they have us available as content area experts to help guide them.

School leaders also must recognize that in order for students to be actively engaged in – and critical consumers of – digital and online information channels, they must have access to technologies and online environments that often are heavily filtered or completely blocked. We can’t help our graduates be citizens and critical thinkers within spaces to which they don’t have access. This is particularly true if we want students to be actively involved within political, scientific, and other digital spaces rather than passive recipients. For instance, teaching online information literacy by pre-selecting a small handful of resources for students to analyze is vastly different from teaching students to navigate and make sense of our vast, complex online information commons.

School leaders also must create safe spaces for teaching and learning about controversial topics. Imagine, for instance, a high school government teacher who asked her students to follow the primary social media channels of the two primary political parties here in the United States. On the Republican side, students could follow GOP websites, Twitter feeds, and YouTube videos and subscribe to conservative blogs such as RedState, HotAir, Instapundit, and Michelle Malkin. On the Democratic side, students also could follow relevant websites, Twitter feeds, and YouTube channels, along with liberal blogs such as Daily Kos, The Huffington Post, Democratic Underground, and ThinkProgress. Sprinkle in a few other sites such as The Hill, Politico, FiveThirtyEight, Fox News, and CNN and we can see how real-time social media could be an incredibly powerful lens through which to view, discuss, and understand government in action, not just as abstract concepts from a dry textbook. I’m not sure how many teachers would be willing to try this, however, given schools’ traditional aversion to anything controversial. Principals and school systems must be willing to buffer a few anxieties in order to enable these kinds of meaningful learning experiences.

Schools also have to stop treating students as ‘digital natives’ who already are knowledgeable about and proficient with technology. Youth fluency with social and gaming technologies may imply certain levels of technology comfort but does not mean that students have the ability to use digital tools in academic- and work-productive ways. Not only is the ‘digital natives’ concept disproven by research, it also seems to grant us permission as educators to avoid the difficult challenge of fostering technology- and information-fluent students because we supposedly have little to teach them. Schools’ reluctance to own this challenge – perhaps because of our educators’ own lack of technology fluency – results in findings like the recent study from Stanford University that showed that students’ current information literacy skills are abysmal.

Finally, school leaders should recognize that those teachers who enable youth to actively interact and create online also are creating opportunities for students to learn essential lessons about responsible participation, sharing, contribution, etiquette, and digital citizenship as natural extensions of their classwork. This approach is far more meaningful and impactful than a few isolated media literacy sessions or digital citizenship lectures. We say that we want engaged citizens and critical thinkers. So let’s do a better job of preparing our students to be thoughtful consumers and active contributors within our new technology-suffused information spaces.

What is your school doing to help students with fake news and filter bubbles?

[cross-posted at Front and Central]

Image credit: Fake news figure, Stuart Rankin

Visibility and reach: Journal articles v. blog posts

Back in 2008 I raised the questions:

Why would anyone who wishes to actually reach educators and hopefully influence change in schools not be blogging?

Also… why haven’t more faculty caught on to this?

Eight years later, I thought that I would share a couple of recent tables that I made in order to illustrate this point further. The first table is the number of academic citations that I have received on the top 20 things that I have written or created (according to Google Scholar). My citation numbers are decent if not spectacular; they’ve been enough to get me tenure at several of our nation’s top research institutions.

2016-10-27journalarticles

[click here for a larger image]

The second table shows the number of page views and comments that I have received on my top 20 blog posts.

2016-10-27blogposts

[click here for a larger image]

No comparison in terms of reach, visibility, interaction, and (hopefully) impact. The percentage of university faculty members who are blogging – although better than it was 8 years ago – is still incredibly low. We pay the price in terms of public and policymaker awareness of and attention to our work.

Iowa school poverty and report card rankings

My local high school recently was named the top high school in Iowa by Niche.com, a school and college ranking site.

Ames High on Niche com

Today the Iowa Department of Education issued its first-ever school report cards. Ames High School didn’t do as well this time, only managing an overall ranking of Commendable, which is the third-highest report card category. Here are the number of Iowa schools in each of the six possible report card categories:

2015 Iowa School Report Categories 2

For this first year, the Department of Education distributed schools along a normal curve. In future years, the point boundaries for the school report categories will be locked into place and schools will be able to move in and out of the categories. In other words, down the road it is possible that some report card categories may have few or no schools in them.

I downloaded the Department’s school report card data and combined them with its free lunch data. Free or reduced-price lunch percentages often are used as indicators of school poverty. Here is what the free lunch percentage distributions look like for each report card category:

Iowa School Report Card Rankings by Free Lunch Percentage 3

Zero of the 34 Priority schools have less than 33% free lunch eligibility and 30 of the 34 (88%) have more than half of their students who are eligible. In contrast, 27 of the 35 Exceptional schools (77%) have less than 33% free lunch eligibility and only 3 of the 35 (9%) have more than half of their students who are eligible. Here are the median and average free lunch eligibility percentages for each report card category:

Free Lunch Percentage for Iowa School Report Card Categories

Here is the box plot for each school report card category:

Iowa School Report Card Rankings by Free Lunch Percentage 1

Here’s a reminder on how to interpret a box plot:

Interpreting a box plot

Iowa’s school report card results mirror those of other states, which typically show strong negative relationships between overall school report card scores and school poverty levels. So we now have an Iowa school report card system that confirms what we already knew from the peer-reviewed research and from other locations, which is that schools with higher poverty levels tend to do less well on indicators of school success. Whether we will actually do anything about it remains an open question…

Please check over my data and see if I made any mistakes. Also see my copyright policy and feel free to use these data and images as you wish for your own projects!

Option 3: Actually USE the smartphones

Door sign: Cell phone prohibited

Murphy & Beland’s recent study is making the rounds online, particularly among those who are eager to find reasons to ban learning technologies in classrooms. The economists found that banning mobile phones helped improve student achievement on standardized test scores, with the biggest gains seen by low-achieving and at-risk students. Here are my thoughts on this…

The outcome measure is standardized test score improvement. Is that all you care about or do you have a bigger, more complex vision for student learning? For instance, creativity, collaboration, critical thinking, and problem-solving are difficult to assess with a standardized test. Most schools I know didn’t adopt their learning technology initiatives for the sole purpose of test score improvement. (if they did, how sad is that?)

The accepted dichotomy in this study and the media seems to be 1) doing low-level knowledge work while smartphones are banned, or 2) doing low-level knowledge work while smartphones are present (and, presumably, distracting). Neither of these two options addresses the fact that decontextualized, low-level work isn’t very interesting or engaging to many (most?) students, particularly those who already find that traditional schooling doesn’t meet their needs very well. So, faced with the opportunity to do something else, many students do. Youth today aren’t any different than when we were young and adults made snarky, woeful comments about us. They just have different opportunities and resources. How many times were you bored in high school? Lots, so admit that if you’d had access to a smartphone or your friends on Facebook back then, you would have turned that way too. I know that I sure would have. Let’s stop blaming students and/or demonizing technology as an evil succubus and address the real problem, which is disengaging learning environments. The solution to that problem is not to try and force students to pay attention to and comply with our boring lessons. That’s not teaching students ‘grit.’ That’s an indictment of our failure to differently imagine learning and teaching.

How about a third option, that of doing higher-level learning and USING the smartphones to help with that? That sounds pretty good to me. Why isn’t this ever brought up as an option to be considered?

Image credit: Cell phone prohibited, SmartSign

Principal 2.0 (new book!)

Principal 2.0

Remember last September when I made available for free my most recent book chapter, Supporting Effective Technology Integration and Implementation?

Well, that chapter’s still available to you for free, but I’m pleased to announce that the rest of the book is now available as well. Featuring a diverse set of authors, perspectives, and topics, there’s something for everybody in Principal 2.0: Technology and Educational Leadership.

Here are the chapter titles and authors (including all 4 CASTLE directors!):

  • Foreword: Ready Set Go! with Educational Technology, Governor Beverly Perdue
  • Introduction: Making the Case for Principal 2.0, Jennifer Friend and Matthew Militello
  • Augmenting Educational Realities, John Militello
  • The Role of For-Profit Firms in the Educational Ecosystem, Michael J. Schmedlen
  • Generation X Meets Generation Y: Reflections on Technology and Schooling, Jennifer Friend and Alexander David Friend
  • Zen and the Art of Technology in Schools: Multigenerational Perspectives, Matthew Militello, Ronald Militello, Dominic Militello, Luke Militello, and Gabriel Militello
  • Digital Storytelling for Critical Reflection: An Educational Leadership Story, Francisco Guajardo, Miguel A. Guajardo, John A. Oliver, Mónica M. Valadez, and Mark Cantu
  • Engaging Youth Voice: Collaborative Reflection to Inform School Relationships, Processes, and Practices, Christopher Janson, Sejal Parikh, Jacqueline Jones, Terrinikka Ransome, and Levertice Moses
  • There’s an App for That: 50 Ways To Use Your iPad, April Adams and Jennifer Friend
  • Student-Owned Mobile Technology Use in the Classroom: An Innovation Whose Time Has Come, Tricia J. Stewart and Shawndra T. Johnson
  • Affective Learning Through Social Media Engagement, David Ta-Pryor and Jonathan T. Ta-Pryor
  • The Central Texas Community Learning Exchange Digi-Book: Fostering School and Community Engagement Through the Creation of a Digital Book, Lee Francis, IV, Mónica M. Valadez, John A. Oliver, and Miguel A. Guajardo
  • Leaders Online: Enhancing Communication with Facebook and Twitter, John B. Nash and Dan Cox
  • Balancing Effective Technology Leadership with Legal Compliance: Legal Considerations for Principal 2.0, Justin Bathon and Kevin P. Brady
  • Connected Principals: In Pursuit of Social Capital via Social Media, Candice Barkley and Jonathan D. Becker
  • School Leaders’ Perceptions of the Technology Standards, Matthew Militello and Alpay Ersozlu
  • Supporting Effective Technology Integration and Implementation, Scott McLeod and Jayson W. Richardson.

A big thanks to Drs. Matt Militello and Jennifer Friend for inviting me to be a part of this publication. Happy reading!

State department educational policy advocacy: “Evidence-based” or puffery?

I teach. What's your superpower?

One of the Iowa Department of Education’s favorite phrases is ‘evidence-based,’ as in “schools and educators should be adopting ‘evidence-based’ practices.” That makes sense on its surface, right? So when the DE issued a press release today touting two Iowa districts that had adopted the National Institute for Excellence in Teaching’s (NIET) Teacher Advancement Program (TAP) – which focuses on teacher leadership and performance-based teacher compensation – I started digging around in Google Scholar for research about the program. I don’t live in the world of teacher leadership/compensation and thus don’t know much about TAP other than that it appears to support the DE’s policy proposals this legislative session (and that Dr. Brad Buck, an amazing school leader that I greatly respect, is the superintendent of one of the two districts). For all I know, TAP could be amazing or it could be puffery

What did I find?

And, of course, on its web site I found research from NIET itself claiming that TAP is awesome. Indeed, there’s a plethora of publications on TAP’s web site from NIET; agencies such as the Milken Family Foundation, the Joyce Foundation, or the Gates Foundation that have funded TAP; and ideologically-oriented educational organizations. But despite TAP’s claims of ‘a decade-long track record of growth and success in raising student achievement in high-need schools‘ (page 19), the evidence from independent researchers is decidedly more mixed and leans much more pessimistic when it comes to student learning outcomes.

So is TAP ‘evidence-based’ or puffery? Given the greater preponderance of negative findings by outside, independent scholars, it appears to be extremely arguable. Given that uncertainty and its own call for ‘evidence-based’ practices, should DE be basing its 2013 education reform package on the TAP model? (and should it be touting the model via press releases to the Iowa public and media, which know even less about all of this than I do?)

We already know that DE is willing to ignore decades worth of peer-reviewed research when it conflicts with its policy advocacy (see, e.g., recent incidents regarding 3rd grade retention and cutting elementary school recess). I don’t know if TAP falls into this category of willful blindness or not. But I am wondering if the studies listed above were ever presented to those in decision-making positions so that they could make a truly informed decision.

I hope that the initiatives go well for the two Iowa districts that are trying TAP. Their educators are going to invest a great deal of time, energy, and money in the model. Hopefully they will see the results for which they are striving. Until I see further independent research supporting TAP (send it if you’ve got it; maybe I missed some!), right now I’m less sanguine about whether we should be basing tens of millions of dollars statewide to implement the model here in Iowa.

Got any thoughts on this?

[Note that there’s also a bigger question here that’s always worth considering: What should we do if/when our own governmental agencies fail to apply the same standards to themselves that they wish to apply to us?]

Image credit: What’s your superpower?

Text complexity in the early grades: Shooting ourselves in the foot?

Cozy reading spot

Here are some quotes from the most recent issue of Educational Researcher regarding text complexity in the early grades, one of the hallmark pushes of the Common Core State Standards:

the CCSS text complexity standards for Grade 3 appear to be aspirational, much like the No Child Left Behind Adequate Yearly Progress targets (Shepard, 2008). The small set of studies that have examined text complexity over time does not show that text complexity at Grade 3 has deteriorated. Neither is there evidence that the accelerated targets in the primary grades are necessary for high school graduates to read the texts of college and careers. (p. 47)

AND

Another potential indirect effect on students may be their motivation and engagement. The engagement of reading among American students is already low, as indicated by a 2001 nationally representative sample of fourth graders from 35 countries that ranked the United States 33rd in an index of students’ motivations for reading (Mullis, Martin, Gonzalez, & Kennedy, 2003) and 35th out of 35 countries in the revised index of attitudes toward reading (Twist, Gnaldi, Schagen, & Morrison, 2004). At present, there is research indicating that motivation decreases when tasks become too challenging and none that indicates that increasing challenge (and potential levels of failure) earlier in students’ careers will change this dismal national pattern of disengagement with literacy (Guthrie, Wigfield, & You, 2012). (p. 48)

AND

Will the intended outcomes of higher levels of literacy for all students be realized by setting the bar arbitrarily at third grade? Our review suggests that the unintended negative consequences could well outweigh the intended positive outcomes. (p. 49)

AND

Increasing the pressure on the primary grades – without careful work that indicates why the necessary levels are not attained by many more students – may have consequences that could widen a gap that is already too large for the students who, at present, are left out of many careers and higher education. How sadly ironic it would be if an effort intended to support these very students limited their readiness for college and careers. (p. 49)

Hiebert, E. H., & Mesmer, H. A. E. (2013). Upping the ante of text complexity in the Common Core State Standards: Examining its potential impact on young readers. Educational Researcher, 42(1), 44-51.

Image credit: “Cozy” reading spot