Another dot in the blogosphere?

Evaluating evaluations

Posted on: March 16, 2016

This is the second part of my thoughts on flawed evaluation of instruction and professional development. This was yesterday’s prelude.

Most training and professional development outfits conduct a survey at the end of a session. This is typically a Kirkpatrick Level 1 form otherwise known as the “smiley sheet”. These forms collect immediate self-reported impressions from the participants of the experience and the provider.

Level 1 forms suffer several weaknesses, among them:

  • Unreliable self-reporting (inconsistency over time)
  • Invalid self-reporting (poorly phrased or misinterpreted questions)
  • Middling scores from disinterested or undecided participants
  • High scores from participants erring on the side of caution
  • Inconsistent design over time or between interventions (e.g., 4 through 7-point Likert-like scales)
  • No or low correlation to other levels of evaluation

Charlatans also know how to take advantage of the weaknesses of such forms. They create a show to wow and thus manipulate the Level 1 feedback. If unethical vendors or instructors are invited to design such forms, then the questions can be manipulated by vendors to favour positive responses.

Even if a form is outside their control, charlatans can focus on behaviours that are measured (e.g., content delivery or speaking ability) and ignore unlisted ones (e.g., risk-taking or promoting critical thinking).

Now this is not to say that behaviours like skilful content delivery and a velvety tongue are not important. However, it is easy to fool people into thinking they are getting a lot of content with persuasive rhetoric.

The larger question is whether the learning experience is meaningful and actionable. Level 1 forms are rarely designed to go beyond initial impression, what feels good, and what is easy to measure.

I do not conduct Level 1 evaluations of my workshops or seminars partly because the organiser does them and also because I know they do not work in isolation.

When I was invited to conduct a long-running series of seminars and workshops for an organisation, I was also required to design my own evaluation reports. Rather than design a Level 1 form, I decided to do something quite different.

I will share the design principles of this evaluation strategy tomorrow.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Click to see all the nominees!

QR code

Get a mobile QR code app to figure out what this means!

My tweets


Usage policy

%d bloggers like this: