Eight Tips for Designing a Training Evaluation Questionnaire

Following my recent blog Training Evaluation…How to Measure what Matters, now here’s a look at using questionnaires as a
training evaluation tool.

There are several decent online surveying tools for designing and distributing questionnaires and analysing results. The easiest is SurveyMonkey, but you may need more than the free version.

The prevalence of these tools has encouraged many people with no experience in writing or interpreting research to have a go, hence the volume of overly long surveys with ambiguous and over complicated questions and unbalanced rating scales that ambush our inboxes.

So where do you start to get your training evaluation questionnaire right?

1. Start with the end in mind

Start with the end goal. That means you should:

• Clarify the training programme objectives (see Training Evaluation…How to Measure what Matters);
• Determine which issues the training evaluation questionnaire must address and don’t get carried away with ‘nice to haves’;
• Clarify who is going to use the survey and see the results;
• Confirm what decisions will be made as a result.

2. Choose the right training evaluation methods

Don’t assume a questionnaire is automatically the right training evaluation tool. They are great for many things, in particular:

• Facts and figures, rankings, average scores and preferences;
• How many, how often, when, where and who;
• Comparisons (before vs after, or one group vs another).

But if you are making complex decisions, you may need a statistically significant sample size. And that requires a very high response rate. Depending on the objectives of your research, you may find qualitative research is more appropriate.

Qualitative research from in-depth structured interviews, focus groups, observation etc, can be a useful steer if you cannot access the wider audience. It can also be a pre-cursor to a questionnaire. You should also opt for qualitative research to:

• Deal with sensitive issues;
• Explore feelings and attitudes;
• Look for ideas and creative solutions;
• Answer the ‘whys’ and uncover truths.

You may want to consider different training evaluation methods for different stakeholders, eg learners, their managers, peers, L&D managers, trainers etc, to give a full view of the programme’s outcomes.

3. Structure your survey for user-friendliness

A well designed training evaluation questionnaire flows logically and comfortably. To achieve this:

• List all essential information required across all stakeholders;
• Group information by topic;
• Sequence topics starting with broader issues before drilling down into more detail;
• Aim for consistency and don’t jump between topics;
• Keep sensitive or more challenging personal questions at the end.

4. Find out who your respondents are

Classification (or profiling) questions about your respondents help you do deeper analysis of age, gender, location, employment status, length of service etc. Classification questions are usually at the end of the survey with a couple of exceptions:

• If you need to determine whether a respondent is eligible to take the survey (eg you may exclude very recent joiners);
• If you want to route the respondent based on their answers (eg staff in different divisions may answer a different set of questions).

5. Ask the right questions

When deciding what to ask, think about how the results will be used. I recommend including questions about all four levels of the Kirkpatrick model. You could even sequence the questionnaire through the levels, eg:

• Was the pace of the course right? (Level 1 – quality control);
• Indicate the level of your skills / knowledge before and after training (Level 2);
• Rate how confident you feel about xyz following training (Level 2);
• Have you applied the learning since the course, if so give examples of how (Level 3);
• In what ways has the training enabled you to make a positive benefit, eg saved time / cost, increased revenue, improved customer retention, increased your efficiency, reduced accidents? (Level 4).

6. Be consistent

With rating scales, you may wish to label each point of the scale, but do make sure the wording is ‘balanced’, ie you use the same terminology throughout the scale, and the mid-point is truly neutral. The example below might help.

On the following scale, please show how much you agree or disagree with the statement:

I understand rating scales very well:

• Strongly disagree;
• Disagree;
• Neither agree or disagree;
• Agree;
• Strongly agree.

7. Test it

Piloting your questionnaires will ensure questions are unambiguous and easy to complete. You can also time how long it takes someone unfamiliar with the survey to complete it. Best attention span is under 15 minutes and you may be surprised how short the survey needs to be to fit within that. Use ‘word’ selections, smiley faces or other images to break up the monotony of a standard rating scale.

8. Explain the relevance of your survey

Last, but by no means least, if you are asking people to give up their time to complete your survey, it is only fair they understand why they are doing it.

I hope these pointers help you improve your surveying. Don’t panic if you find some of the research techniques daunting – a little friendly advice could save a lot of hassle and dramatically improve the final results. Do feel free to add a comment at the bottom. I’d love to know what tools and techniques you are using – and happy to help with any issues or questions.

Liz Barnes

Liz is a qualified trainer who specialises in the commercial aspects of management training. With extensive experience in both the private and public sector, she has also complemented her facilitation role with consultancy and lecturing in management and marketing.