Posted March 16, 2016on:
This is the second part of my thoughts on flawed evaluation of instruction and professional development. This was yesterday’s prelude.
Most training and professional development outfits conduct a survey at the end of a session. This is typically a Kirkpatrick Level 1 form otherwise known as the “smiley sheet”. These forms collect immediate self-reported impressions from the participants of the experience and the provider.
Level 1 forms suffer several weaknesses, among them:
- Unreliable self-reporting (inconsistency over time)
- Invalid self-reporting (poorly phrased or misinterpreted questions)
- Middling scores from disinterested or undecided participants
- High scores from participants erring on the side of caution
- Inconsistent design over time or between interventions (e.g., 4 through 7-point Likert-like scales)
- No or low correlation to other levels of evaluation
Charlatans also know how to take advantage of the weaknesses of such forms. They create a show to wow and thus manipulate the Level 1 feedback. If unethical vendors or instructors are invited to design such forms, then the questions can be manipulated by vendors to favour positive responses.
Even if a form is outside their control, charlatans can focus on behaviours that are measured (e.g., content delivery or speaking ability) and ignore unlisted ones (e.g., risk-taking or promoting critical thinking).
Now this is not to say that behaviours like skilful content delivery and a velvety tongue are not important. However, it is easy to fool people into thinking they are getting a lot of content with persuasive rhetoric.
The larger question is whether the learning experience is meaningful and actionable. Level 1 forms are rarely designed to go beyond initial impression, what feels good, and what is easy to measure.
I do not conduct Level 1 evaluations of my workshops or seminars partly because the organiser does them and also because I know they do not work in isolation.
When I was invited to conduct a long-running series of seminars and workshops for an organisation, I was also required to design my own evaluation reports. Rather than design a Level 1 form, I decided to do something quite different.
I will share the design principles of this evaluation strategy tomorrow.