Performance Focused Smile Sheets: Applied

How many training sessions have you gone to where you received a one sheet evaluation form that asked you to rate your instructor, the course, the room, the chair and the snacks provided at ten a.m.? Chances are you have filled out a few of these over time. In my own experience I have probably filled out hundreds and after a while, there is a tendency to just tick off “strongly agree” on everything, especially if it’s getting close to supper time!

Once in awhile, a new method comes along that radically changes the way we do things. Fire, the wheel, smartphones… you get the idea. Have you heard about Dr. Will Thalheimer’s book Performance Focussed Smile Sheets: A Radical Rethinking of a Dangerous Art Form? Will is one of my top go-to guys in evidence-based performance improvement and for myth busting methods being used in the field that aren’t so evidence-based.

Will explains why the current design of end of training evaluations are actually counter-productive, and sums it up nicely with this list of nine points:

1. They are not correlated with learning results.
2. They don’t tell us whether our learning interventions are good or bad.
3. They misinform us about what improvements should be made.
4. They don’t enable meaningful feedback loops.
5. They don’t support smile-sheet decision making.
6. They don’t help stakeholders understand smile-sheet results.
7. They provide misleading information.
8. They hurt our organizations by not enabling cycles of continuous improvement.
9. They create a culture of dishonest deliberation (Thalheimer, 2016, Kindle Locations 137-143).

That’s just in the book’s introduction! Will uses the rest of the book to show us all a better way for “creating smile sheets that will actually help us gather meaningful data— data that we can use to make our training more effective and more efficient” (Thalheimer, 2016, Kindle Locations 2646-2647) by targeting training effectiveness and actionable results.

Actionable results are where I am going to focus the rest of my discussion or this would be a really long post! I recently conducted a session of the International Society for Performance Improvement and Dr. Roger Chevalier‘s workshop “Improving Workplace Performance” for a group of 20 school board managers. The post workshop survey of 19 questions was delivered electronically using SurveyMonkey. Within a week I received 16 responses for a completion rate of 80%

Here are the results from question #1 which Will calls “The world’s best smile-sheet question.”  By asking this question we are getting a measure of potential for the trainee’s improvement back on the job.

37.5% responded “I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.”

62.5% answered “I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.”

Dr. Thalheimer provides a rubric or set of standards in the book to measure the responses for each question. The standards for question 1 are shown in Table 1 below.

Not bad, but not what I can accept either. Two-thirds of the respondents felt they would be able to employ the methods taught in the workplace with more practice. One-third has a general awareness but won’t be able to apply what they learned. This is a one-day workshop that covers a lot of ground and arguably the goal is to raise awareness of performance improvement. There is also a follow on “at-work” component that the learners can do to further increase their skills and earn a certification if they choose. My goal is to have all the learners choosing C or D. Clearly, I have some work to do in the design and delivery departments for this offering.

The result above made me question if the learners had enough hands-on practice with the case study and the exercises. That takes us to question 9 shown below.

 

 

The averaged response was 54%. Will believes (and I agree) that the absolute minimum for time devoted to practice is 35%. Given the  number of practical exercises, I think this number needs to be higher… in the 65% range, so that gives me some quantifiable results and work to make changes to the design before the next session. More practice – less lecture. Check!

One final example. Have you heard of spaced learning theory? Casebourne (2015) provides a good overview of the body of research that suggests that by spacing learning over time, people learn more quickly and remember better. Will has designed questions such as #11 below to measure spacing.

The results were interesting because one respondent apparently went back to the training facility the following day! Overall, it seems that the spacing designed into the workshop was effective and 69% of the respondents recognized that topics were covered more than once. As noted above, the learners do have the opportunity to apply what they learned back on the job and submit it to ISPI to earn a certification which is another spacing strategy, but one I have little control over.

In order to get a measure of actual performance improvement for question #1, and to accurately measure the spacing effect, I will need to conduct the survey again after the learners have had sufficient time to apply the skills they were taught on the job. That’s still to come.

If you are still using level one evaluations or smile sheets that ask if the learning was fun, if the learner liked the instructor and the facilities were comfortable, it’s time to re-think your approach. If you attend a training session and still receive those old style smiley sheets, you might also ask yourself how effective the training design really was. I hope this example has shown you enough evidence to convince you there is a better way. If it was – please share it with your friends and colleagues. Heck – share it with your enemies, they might become your friends!

References

Casebourne, I. (2015) Spaced learning: An approach to minimize the forgetting curve. Retrieved from https://www.td.org/insights/spaced-learning-an-approach-to-minimize-the-forgetting-curve 06 December, 2017.

Thalheimer, W. (2016). Performance-focused smile sheets: A radical rethinking of a dangerous art form (). Work-Learning Press. Kindle Edition.