On Learning, Testing, and Being Tone Deaf

There are plenty of reasons to believe that multiple-choice and true-false tests are among the worst ways to measure whether learning is successful; in the best of circumstances, they tend to measure only the lowest levels of learning achieved, and in the worst of situations they leave respondents without any acceptable responses from which to choose. Yet we seem to remain tone deaf to concerns and criticisms voiced by educators for decades and continue to rely on them.

From Skley’s Flickr photostream at http://tinyurl.com/machqej

The multiple-choice/true-false approach is pervasive in much of what we encounter day to day within and outside of the world of learning. Surveys often incorporate these methods to make tabulation of the results easy—after all, it can easily be argued, we can’t afford to engage in more personalized labor-intensive methods of collecting data (e.g., reading short- or essay-length responses)—although there are signs that mechanizing the testing and grading process in ever-more sophisticated ways is increasingly becoming possible.

But relying on mechanical methods so obviously produces mechanical results that I believe we need to question our own assumptions, be honest about what those assumptions are producing, and seek better solutions than we so far have managed to produce.

A couple of recent experiences helped me understand the frustrations and weaknesses of multiple-choice/true-false testing at a visceral level: taking two very good and very different massive open online courses (MOOCs), and responding to a survey that relied solely on multiple-choice responses which, because they were poorly written, didn’t provide any appropriate responses for some of us who would have been very willing to participate in the project to which the survey was connected.

Let’s look at the MOOCs first: #etmooc (the Educational Technology & Media MOOC developed and facilitated by Alec Couros and his wonderful gang of “conspirators” earlier this year) and R. David Lankes’s “New Librarianship Master Class” (a MOOC developed and delivered under the auspices of the University of Syracuse School of Information Studies). #etmooc, as a connectivist MOOC (where learning, production of learning objects including blog posts and videos demonstrating the skills that learners were acquiring, and online connections between learners were among the central elements and testing was nonexistent) nurtured development of a community of learning that continues to exist among some of the participants months after the course formally ended. The New Librarianship Master Class, grounded in the more traditional academic model of documenting specific learning goals, relied more on standardized testing to document learning results and offered less evidence that learners were using what they were learning.

The frustrations that some of us mentioned after struggling with standardized questions that didn’t really reflect what we had learned and what we would be capable of doing with that newly-gained knowledge resurfaced for me last night as I was trying to complete in an online research survey. Facing a series of multiple-choice questions, I quickly realized that whoever prepared the options available as responses to the survey questions had underestimated the range of experiences the survey audience had—was, in fact, tone deaf to the nuances of the situation. Opting for results that could be scored mechanically rather than requiring any sort of human engagement in the tabulation process, the writer(s) forgot to include an option for “other” to catch any of us whose experiences didn’t fit any of the options described among the possible multiple-choice responses—which, of course, meant that at least a few of us who might otherwise have been willing to provide useful information abandoned the opportunity and turned to more rewarding endeavors. (And no, there wasn’t an opportunity to simply skip those questions-without-possible-answers so we could stay involved.)

Abandoning a potentially intriguing survey had few repercussions other than the momentary annoyance of being excluded from an interesting project. Being forced to respond to standardized tests that don’t accurately document a learner’s level of mastery of a subject is obviously more significant in that it can affect academic or workplace advancement—and it’s something that is going to have to be addressed in those MOOCs that don’t take the connectivist approach—which raises a broader question that need not be answered within the confines of pre-specified options on a multiple-choice test: why are we not advocating for more effective ways and resources to encourage and document learning successes in both onsite and online settings? Expediency need not be an excuse for producing second-rate results.

Workplace Learning and Performance: Optimism and Responsibility

Learning executives across the United States are more optimistic about the training industry than at any other time since ASTD (the American Society for Training & Development) began issuing its quarterly Learning Executives Confidence Index highlights reports two years ago, the latest summary shows.

The news is not particularly astonishing; the project began around the same time the worst recession most of us have faced began. It does, however, reflect the improvements many of us have been noticing over the past year in workplace learning and performance opportunities.

Nine out of ten of the 354 respondents to the invitation-only survey “expect the same or better performance for their [workplace learning and performance] industry in the next 6 months,” and seven out of ten expect “moderate to substantial improvements” (p. 5).

More than four out of ten respondents anticipate “increased expenditures on outsourced or external services to aid in the learning function in the coming months of 2011. Outsourced or external services include such expenses as consultation services, content development, content and software licenses, and workshops and training programs delivered by external providers” (p. 8).

Two-thirds of the respondents think the use of e-learning will “moderately or substantially” increase during the next six months, and they see a similar increase in the use of Web 2.0 technology—again, not surprising given the number of social networking tools such as Twitter, Skype, blogs, and podcasting tools used as vehicles for delivery of learning opportunities.

This is far from insignificant; workplace learning and performance, according to ASTD’s “2010 State of the Industry Report,” is a $125.8 billion industry annually (p. 5 of the “State of the Industry Report”). It’s an important part of our overall commitment to lifelong learning. And, as ASTD representatives playfully note, it’s part of an effort designed to “create a world that works better.”

In spite of the encouraging news documented in the quarterly Confidence Index report, there is no time for complacency here. The way we learn and the way we offer learning opportunities is changing in response to the availability of online tools, and continuing economic pressures hinder learners’ opportunities to travel to attend face-to-face learning sessions (p. 9 of the Confidence Index report). There are also plenty of examples of stultifyingly ineffective face-to-face and online learning offerings that diminish rather than encourage learners’ enthusiasm, as any of us who regularly attend training sessions can confirm.

On the other hand, there are plenty of organizations like the more than 125 ASTD chapters across the United States and the national society itself that offer learning opportunities for trainer-teacher-learners interested in improving our knowledge, skills, and ability to meet workplace learning and performance needs.

The responsibility to engage in actions that would merit and nurture the optimism expressed by those 354 learning executives who contributed to the 2011 First Quarter Learning Executives Confidence Index report remains firmly in our hands.