Real problems versus test problems

Real problems versus test problems

Robert Sternberg said:

The characteristics of real-world problems are entirely different from the characteristics of problems on standardized tests. Standardized test problems are mostly multiple choice or short answer and have a right or wrong answer. Real problems require extended answers; there is no perfect answer, and sometimes, not even a very good one. Standardized test problems are decontextualized, emotionally bland and have no real-life stakes. Real-world problems are highly contextualized, emotionally arousing and may have high stakes. Standardized test problems are solved quickly and then you are done; real-life ones often take a long time and, after you think you have solved them, often come back.

 

Most important, real-world problems require you actively to deploy your intelligence — to decide seriously to use it. Standardized tests measure an inert form of intelligence — one that may exist in your head somewhere but is rarely actually put into real-world use. Intelligence is not just about an inert ability to take tests; it is about the active deployment of that ability to solve problems of life.

 

 

Is adaptive intelligence really important? Well, you be the judge. Which skill is more important for the great majority of students in college once they have graduated: the ability to solve artificial verbal and math problems or, alternatively, to address and try to solve problems of global climate change, air and water pollution, global pandemics, bacterial resistance to antibiotics, gun violence against schoolchildren (other than the usual pathetic “our thoughts and prayers are with them”), and the return of would-be autocrats to declining democracies?

There’s usually a difference between academic work and authentic work, and that difference is important when schools talk about the ‘critical thinking’ and ‘problem solving’ opportunities that they offer students. Looking at you, Advanced Placement, International Baccalaureate, “advanced classes,” “honors courses,” etc…

Your thoughts?

If you want to graduate, pass the test or grovel to the school board

If you want to graduate, pass the test or grovel to the school board

About ten years ago when I was in Iowa, a middle school principal decided that her students weren’t trying hard enough on the state tests. So she set up a fun end-of-year field trip to the amusement park and told the students that whomever didn’t do their best during assessment season couldn’t go. I asked her how many students didn’t get to go, and she said less than a dozen. I asked her how she decided who didn’t do their best, and she said, “We can tell when we walk around during testing sessions.” I asked her how she thought those extremely few students felt as they were singled out and left behind at school while everyone else in the school was having fun. She didn’t care about those students’ well-being. All she cared about was the message that she thought she was sending those few students about taking their academics seriously. I invited her to consider that perhaps that wasn’t the message that they were receiving. She didn’t hear me. As you can imagine, it was a pretty depressing discussion.

Fast forward to 2023 and here we go again, also in Iowa. The Maple Valley – Anthon Oto Community Schools have decided that their high school students aren’t taking the Iowa Statewide Assessment of Student Progress (ISASP) seriously enough. So they are now tying ‘proficiency’ on the 11th grade ISASP to high school graduation. Students who aren’t ‘proficient’ in all four ISASP areas (yes, all four!) must either then show ‘proficiency’ on NWEA’s MAP assessments (which isn’t really measured by MAP; it has to be imputed) or write a letter to the school board explaining why they should be allowed to graduate anyway. See the images below for the letter to families.

So if you’re a student in this community who is a poor test taker, you can’t graduate – even if you’ve passed all of your required courses – unless you somehow show ‘proficiency’ on all four of the standardized tests anyway or grovel to the school board and hope that it is merciful. This is terrible and has absolutely no place in education.

Your thoughts?

MVAOCOU 01

MVAOCOU 02

Judging school success by test scores. And only test scores.

John Merrow said:

Apparently it’s pretty simple for the folks administering the Broad Prize in Urban Education: Successful School Reform boils down to higher test scores. There is no public sign that anyone at the Foundation is questioning whether living and dying by test scores is a sensible pedagogy that benefits students. There is no public evidence that anyone at the Foundation has considered what might happen if poor urban students were exposed to a rich curriculum and veteran teachers, which is essentially the birthright of students in wealthy districts. Just the dismal conclusion that traditional districts are incapable of reform, followed by its decision to double down on charter management organizations, despite the truly offensive record of some of them of excluding special needs children and driving away students who seem likely to do poorly on standardized tests.

via https://themerrowreport.com/2017/05/12/the-canary-in-the-mine

Bribing children to take our tests

Bribe

It’s standardized testing season again in American schools. And that means it’s also time for many schools to bribe and punish their children into submission because those tests are ones they don’t want to take.

Over the past couple of decades, the political stakes attached to standardized testing have accelerated greatly. So too have teachers’ and administrators’ concerns about their schools’ scores. As a result, there now exists a staggering range of ‘motivational’ efforts that attempt to get students in a positive mindset about testing. For instance, a search for ‘test prep rally’ on YouTube returns over 250 videos of school plays, lip synced songs, and ‘Slam the Exam’ concerts. On Pinterest and at Teachers Pay Teachers, educators can download and attach to candy over 40 different cute, motivational phrases such as ‘You were MINT to succeed’ or ‘You’re a STARBURST of knowledge’ or ‘It’s CRUNCH time. Show what you know!’ At Minds in Bloom, schools can get tips about costuming, audience participation, songs, dances, cheers, jokes, skits, videos, and slide presentation decks for their own test prep rallies. They also can hire the Morris Brothers to perform original songs and share their testing strategies and stress reduction tips. Or they can tap into the numerous other web sites that will help them implement raffles, revise song lyrics, make posters with test taking tips, and stage Are You Smarter Than Your Teachers? game shows.

More troublesome are the post-test ‘celebrations of learning’ that are available only to certain children. A Colorado school made the news recently for its plans to reward those students who show up for every testing day and ‘try their hardest’ (one can only imagine how that will be measured), despite state laws that allow students to opt out of state testing without penalty. As Alfie Kohn reminded us long ago, the withholding of a reward is most certainly a punishment, particularly in the eyes of young children. Is it kind and sensible for educators to preclude from the fun those children who exercised their legally-protected rights? Similarly, I know of a school in Iowa that kept half a dozen of its eight hundred students back from its trip to the video game / bowling / laser tag center because the principal felt that they hadn’t given their best effort on the state exams. Do you think those students ‘learned their lesson’ and will ‘try harder’ next year? Or will they merely be resentful and see the punishment as just another example of their school’s lack of support for their learning challenges?

The justification in all of these cases is that the tests are ‘important,’ that the schools can face potential penalties for poor performance or lack of participation, and that students need to take the assessments seriously. But how seriously should the students take them? After all, our children don’t get any noticeable, tangible benefits from these exams. It’s not as if they can get the questions afterward, see what they missed, get timely feedback on how they did, and get learning assistance from their teachers. All they receive is a meaningless-to-them set of numbers, bar charts, and percentile rankings 4 to 6 months later, typically in their next year of schooling when it’s much too late to really be helpful. And if they attempt to discuss in any way what the questions were and how they think they should have solved them, they get in trouble for ‘cheating’ or ‘violating test security.’ Moreover, the testing windows are artificial events that get inserted into – and usually disrupt the pacing and flow of – the school year. They also often suck up all of the school computers and Internet bandwidth for weeks on end, taking away technology-enriched learning opportunities.

Let’s face it, these assessments are rarely seen by children as a natural outgrowth of their learning. Instead, they are high pressure, high stress activities that are forced upon them by their school systems. These tests are for adults, plain and simple. And while some students may be eager to please their teachers or ‘help out’ their school, it’s hard to argue with those who weigh differently where they want to place their time, effort, energy, and attention. After all, if we have to bribe or punish our students into taking our exams, that’s probably a sign that we need more meaningful assessments…

What do test prep and student ‘motivation’ efforts look like in your school?

[A modified version of this post is at TrustED under the title, Test prep rallies, ’slam the exam’ concerts, and other testing season follies

Image credit: BRIBE, Alpha Bravo Foxtrot

Are our definitions of ‘college readiness’ too high?

B

David Freedman said:

The College Board has suggested a “college readiness benchmark” that works out to roughly 500 on each portion of the SAT as a score below which students are not likely to achieve at least a B-minus average at “a four-year college” – presumably an average one. . . .

How many high-school students are capable of meeting the College Board benchmark? This is not easy to answer, because in most states, large numbers of students never take a college-entrance exam (in California, for example, at most 43 percent of high-school students sit for the SAT or the ACT). To get a general sense, though, we can look to Delaware, Idaho, Maine, and the District of Columbia, which provide the SAT for free and have SAT participation rates above 90 percent. . . . In these states in 2015, the percentage of students averaging at least 500 on the reading section ranged from 33 percent (in D.C.) to 40 percent (in Maine), with similar distributions scoring 500 or more on the math and writing sections. Considering that these data don’t include dropouts, it seems safe to say that no more than one in three American high-school students is capable of hitting the College Board’s benchmark. Quibble with the details all you want, but there’s no escaping the conclusion that most Americans aren’t smart enough to do something we are told is an essential step toward succeeding in our new, brain-centric economy – namely, get through four years of college with moderately good grades.

via http://www.theatlantic.com/magazine/archive/2016/07/the-war-on-stupid-people/485618

Similarly, ACT estimates that only about 28 percent of recent high school graduates meet its alleged ‘college-readiness benchmarks’ in all four subjects of reading, English, math, and science.

For the record, the 6-year graduation rate in 2013 for first-time, full-time undergraduate students who began at a 4-year postsecondary institution in fall 2007 was 59%. Not all of those graduates have B- averages, of course. But, nonetheless, perhaps these definitions of ‘college readiness’ from SAT and ACT are too stringent?

Image credit: B, Nadège

Michael Bloomberg on testing

Michael Bloomberg

Here are some quotes from Michael Bloomberg about testing students, with my annotations in italics…

  1. “Many companies (including mine) use tests in hiring.” Really? The hiring ’tests’ for your financial software, data, and media company are multiple choice tests of factual recall and procedural regurgitation?
  2. “Students will face tests throughout their life. They must learn to cope with the emotional stress that comes with the experience.” Just curious: Do your workers cry, get stomach aches, or wet themselves when they face emotional stress in your workplace? (like some of our elementary students do at testing time) If so, must be a fun place to work!
  3. “Test-taking is no one’s idea of fun, but it is part of life.” Quick. Name other areas outside of school and college admissions where taking multiple choice exams and writing short, formulaic essays that are graded in 1-2 minutes are a regular part of life.
  4. “In the ultracompetitive global economy, the U.S. is facing a terrible mismatch between high-skill jobs and our labor pool.” How, exactly, do standardized tests of low-level knowledge lead to high-skill jobs? How, exactly, does an emphasis on low-level thinking work foster higher-level thinkers? What’s your theory of action?
  5. “The biggest threat to American might is not any one country or terrorist group. It is our collective unwillingness to confront mediocrity in our schools.” Many of us ARE confronting mediocrity in our schools. We are confronting the mediocrity of our continued emphasis on assessments of low-level thinking work instead of assessments of critical thinking, creative problem solving, effective communication and collaboration, and other higher-level skills.
Your thoughts?

Image credit: Wikimedia

The exam sham

Harvard

Mike Crowley said:

Teachers are being judged and schools rated based on test and exam results. How many kids are getting into Yale and Oxford, Harvard and Cambridge, we are perpetually asked. I have yet to be asked, how many of your students go to the college that is right for them? … how many are pursuing their passions? … how many are leading happy, fulfilling lives and believe that the curriculum was relevant to their daily, real-world challenges? No, we rarely ask the right questions.

via http://crowleym.com/2015/06/21/the-exam-sham-onwards-we-blindly-go

Image credit: Harvard, Anne Helmond

Notice the emphasis on ‘feeding them content’

Christine Willig, President of McGraw-Hill Education, said:

There’s a difference between educational technology – a single video, a single interactive, a single app – and learning science, in which we’re investing in the small pieces of data that show us where a child is at in their learning trajectory, feeding them content in a way that’s powerful and effective for them to move to the next level. 

via https://www.youtube.com/watch?v=NYR47-XJq64#t=01m55s

Test prep works

Bubble test

Sarah Blaine said:

ten years into private practice, I don’t draw on my two months of intensive bar test prep to advise my clients or manage my work. I don’t rely on essay formulas to craft my briefs, and of course I have never encountered an MBE-style multiple choice question. But the thing is… PMBR and BAR/BRI worked. Test prep works. Test prep taught me to immerse myself in the logic of the test-makers, and how to effectively game the system to achieve my goal: a passing score.

The fact that test prep works is what scares me as a public school parent, because as a parent I know that my child’s standardized test scores tell me virtually nothing about whether she’s actually mastered the academic skills she needs for a successful future.

My two months of bar test prep taught me that mass-produced bar prep can successfully raise scores: my MBE score skyrocketed when I left my inquisitiveness, curiosity, and thoughtfulness at the door, and instead immersed myself completely in the test-makers’ logic. I was willing to engage in two months of intensive test-prep because the stakes were so high: I could have lost my new job for failing the bar. Test prep was a means to an end, and it was an end I wanted (passing the bar so I could begin my career as a litigator at a large law firm), so I was willing to spend (my firm’s) money and my time on the commercial test prep courses. Thankfully, though, our (generally tenured) law school professors focused on preparing us for the practice of law, and not on preparing us for a soon-to-be-forgotten standardized test.

But what will my child gain from devoting 9 of her 13 years of public education to test prep? She might become a genius at immersing herself in the logic of the test makers, but will she learn to write purposefully and well? Will she learn to creatively attack a problem? Will she learn empathy and art appreciation and history and how to work as a member of a team?

via http://www.washingtonpost.com/blogs/answer-sheet/wp/2015/04/24/why-its-so-scary-that-test-prep-works

Image credit: Bubble World, Benjamin Chun

Let’s be honest about annual testing

Testing pencils

Let’s be honest: students and parents obtain no tangible benefit from large-scale annual testing. Kids and families give up numerous days of learning time – both for the tests themselves and for the test prep sessions whose sole purpose is to get ready for the tests (and maybe also for the testing pep rally) – and for what? The data come back too late to be actionable. The questions are shrouded in secrecy so that no one has any idea what students actually missed. As Diane Ravitch has noted, given the immense amounts of time, energy, money, and personnel that we expend on our summative assessments, “there’s no instructional gain … [there’s] no diagnostic value.” The tests fail the fundamental rule of good assessment – which is to provide feedback to fuel future improvement – and come at a tremendous opportunity cost.

All of this might be fine – students and families might dutifully and kindly take a few hours or even days out of the school year to support their local school’s desire to get some institutional-level benchmarks (like when I was a kid) – if the stakes currently weren’t so high and the problems weren’t so prevalent (unlike when I was a kid). The use of extremely-volatile, statistically-unreliable data to punish teachers and schools… the misuse of assessment results to fuel anti-public-school political agendas… the billions of public dollars that go into the pockets of testing companies instead of under-resourced classrooms… the narrowing of curricula and the neglect of non-tested subjects… the appropriation of computers for weeks on end for testing instead of learning… the recharacterization of schools as test score factories, not life success enablers… no wonder parents are starting to scream. It’s a miracle that more families aren’t opting out of these tests and it’s awfully hard to blame them if they do.

Our assessment systems are a complete mess right now. As parents experience empty-threat tantrums from policymakers, vindictive ‘sit and stare’ policies from school districts, and testing horror story after horror story, they are rightfully pushing back against testing schemes that offer no learning feedback or other concrete benefits to their children. There are looming battles with governors and the federal government around opt-out policies. Put your money on the parents.

Many educators are still running scared on this front. Most schools are still fearful and compliant. Our inactivity makes us complicit. When do we say ‘enough is enough?’ How bad does it have to get before we stand with our parents and our communities? When do we fight for what’s educationally sound instead of caving in (yet again)?

Image credit: perfect, romana klee