Applying Kirkpatrick Levels within Talent Suite
Level 1: Reaction
This is the most widely recognized and often referred to as the “smile sheet”. Kirkpatrick recommends to create questions that are “learner focused” rather than “instructor focused’.
Some questions:
- Materials allowed me to more easily understand the information.
- The flow of the course agenda provided a clear path for me to understand the topic.
NTS Application:
- Create a standard Course Evaluation in Exam Manager (see How to Create an Exam job guide)
- Assign the Course Evaluation to the learning module in the Module Properties > Assign exam/certification/evaluation step
- Use standard report R318 to export and view the evaluation answers
- Data collected and available for query will include the question, the answer and the user that provided the answer
Level 2: Learning
Answers can be discovered from either the learner or the manager/supervisor perspective. If requested from the learner, then the questions should be “learner focused’ not “instructor focused” as indicated by Kirkpatrick.
Sample questions:
- I believe it will be worthwhile for me to apply what I learned. (Rating)
- I feel confident about applying what I learned back on the job. (Rating)
- What is the importance of applying what you learned on the job? (Essay)
Learner Perspective
- Include these questions within the original Level 1 Course Evaluation thus creating a “Blended Evaluation” (also recommended by Kirkpatrick to avoid survey fatigue)
OR - Create another Course Evaluation in Exam Manager, then
- Create an Assessment Workflow template (see Create Assessment Workflow QRG)
- Decide if both evaluations should be included in Workflow template (eg. Level 1 evaluation available 1 day after course ends, then the Level 2 evaluation available 30/60/X days after course ends)
- Assign to the classroom learning module in the Session Properties, bottom of the Edit Session step in the section entitled Workflow Assignment. Select the Workflow Template from the drop down options.
Manager/Supervisor Perspective
- Create a Supervisor Assessment exam type in the Exam Manager (see Creating an Exam job aid)
- Add this to the Workflow Assessment template previously created.
NOTE: The learner will have this displayed in their Knowledge Center as a requirement, but will be unable to access. The Manager/Supervisor will access this from the Home Page>Workspace>Supervisor Assessment in a section entitled Supervisor Assessments in Assessment Workflows. These are available immediately and do not recognize the post course time restriction identified in the Workflow template.
OR - Create a Course Evaluation whose first question asks the identity of the Learner that the evaluation pertains to (see Exam Questions QRG)
- Create Online learning object (suggested title “Level 2 Evaluation”) with the Course Evaluation as the Launch Properties (see Import Resource QRG).
- Create a Task learning object:
- Module Properties
- DESCRIPTION: The task is to request manager/supervisor to complete a Level 2 Evaluation.
- SIGNOFF OPTIONS: Select Manager (Direct Appraiser) or by Supervisor.
- PRE-REQUISITES: Select the Level 2 Evaluation online learning object
- Session Properties
- MODULE COMPLETION DEADLINE: Edit session step>Module Completion Deadlines choose X days from enrollment (a small time in which the evaluation should begin such as 10 days). Select “Do nothing, deadline was just for reference” so the task remains until signed off.
- EMAIL PREFERENCES SETUP: Select a template in the Task Complete Reminder section. This allows the Learner to send a reminder to the designated signer from the Active Learning list.
- AUTO ENROLL: Completed Courses section. Select the Classroom learning event and # of days after classroom end date (should equate to when the Manager/Supervisor should begin an evaluation – often 30 days).
- Module Properties
- When ready to evaluate their employee (reminded by employee via the Task), the manager/supervisor enrolls in the Level 2 Evaluation Online learning object, completes the evaluation and submits.
- Manager/Supervisor should also sign off the Task as complete. (Workspace>Task Approval)
- Administrators run the standard report R318 for that specific evaluation and emails it to the evaluator
Level 3: Behavior
Did the learner apply what was taught and learned in the classroom learning event? The answers to a follow-up survey can be obtained again from both the learner and the manager/supervisor perspective.
Sample questions:
- How have you used what you learned in training on the job? (Essay)
- What has helped you to implement what you learned? (Essay)
Learner Perspective
- Create a course evaluation in the Exam Manager
- Add this evaluation to the Workflow Assessment template previously created but with an extended date (multiple assessments can be included in a Workflow Assessment).
Manager/Supervisor Perspective
- Create a Supervisor Assessment exam type in the Exam Manager (see Creating an Exam job aid)
- Add this to the Workflow Assessment template previously created.
NOTE: The learner will have this displayed in their Knowledge Center as a requirement, but will be unable to access. The Manager/Supervisor will access this from the Home Page>Workspace>Supervisor Assessment in a section entitled Supervisor Assessments in Assessment Workflows. These are available immediately and do not recognize the post course time restriction identified in the Workflow template.
OR - Create a Course Evaluation whose first question asks the identity of the Learner that the evaluation pertains to (see Exam Questions QRG)
- Create Online learning object (suggested title: Level 3 Evaluation) with the Course Evaluation as the Launch Properties (see Import Resource QRG).
- Create a Task learning object:
- Module Properties
- DESCRIPTION: The task is to request manager/supervisor to complete a Level 3 Evaluation.
- SIGNOFF OPTIONS: Select Manager (Direct Appraiser) or by Supervisor.
- PRE-REQUISITES: Select the Level 3 Evaluation online learning object (Level 3 Evaluation)
- Session Properties
- MODULE COMPLETION DEADLINE: Edit session step>Module Completion Deadlines choose X days from enrollment (a small time in which the evaluation should begin such as 10 days). Select “Do nothing, deadline was just for reference” so the task remains until signed off.
- EMAIL PREFERENCES SETUP: Select a template in the Task Complete Reminder section. This allows the Learner to send a reminder to the designated signer from the Active Learning lists
- AUTO ENROLL: Completed Courses section. Select the Classroom learning event and # of days after classroom end date (should equate to when the Manager/Supervisor should begin an evaluation – often 60 days).
- Module Properties
- When ready to evaluate their employee (reminded by employee via the Task), the manager/supervisor enrolls in the Level 3 Evaluation Online learning object, completes the evaluation and submits.
- Manager/Supervisor should also sign off the Task as complete. (Workspace>Task Approval)
- Administrators run the standard report R318 for that specific evaluation and emails it to the evaluator
Level 4: Results
This level is identifying if the knowledge learned during the classroom event has impact on the business. It can be obtained from either the Learner or the Manager/Supervisor perspective.
Sample questions:
- What early signs of success have you noticed from your efforts? (Essay)
- How has your participation in this program benefited the company? (Essay)
Learner Perspective
- Create a course evaluation in the Exam Manager
- Add this evaluation to the Workflow Assessment template previously created but with an extended date (multiple assessments can be included in a Workflow Assessment).
Manager/Supervisor Perspective
- Create a Supervisor Assessment exam type in the Exam Manager (see Creating an Exam job aid)
- Add this to the Workflow Assessment template previously created.
NOTE: The learner will have this displayed in their Knowledge Center as a requirement, but will be unable to access. The Manager/Supervisor will access this from the Home Page>Workspace>Supervisor Assessment in a section entitled Supervisor Assessments in Assessment Workflows. These are available immediately and do not recognize the post course time restriction identified in the Workflow template.
OR - Create a Course Evaluation whose first question asks the identity of the Learner that the evaluation pertains to (see Exam Questions QRG)
- Create Online learning object (suggested title: Level 4 Evaluation) with the Course Evaluation as the Launch Properties (see Import Resource QRG).
- Create a Task learning object.
- Module Properties
- DESCRIPTION: The task is to request manager/supervisor to complete a Level 4 Evaluation.
- SIGNOFF OPTIONS: Select Manager (Direct Appraiser) or by Supervisor.
- PRE-REQUISITES: Select the Level 4 Evaluation online learning object (Level 4 Evaluation)
- Session Properties
- MODULE COMPLETION DEADLINE: Edit session step>Module Completion Deadlines choose X days from enrollment (a small time in which the evaluation should begin such as 10 days). Select “Do nothing, deadline was just for reference” so the task remains until signed off.
- EMAIL PREFERENCES SETUP: Select a template in the Task Complete Reminder section. This allows the Learner to send a reminder to the designated signer from the Active Learning list.
- AUTO ENROLL: Completed Courses section. Select the Classroom learning event and # of days after classroom end date (should equate to when the Manager/Supervisor should begin an evaluation – often 6 months+).
- Module Properties
- When ready to evaluate their employee (reminded by employee via the Task), the manager/supervisor enrolls in the Level 4 Evaluation Online learning object, completes the evaluation and submits.
- Manager/Supervisor should also sign off the Task as complete. (Workspace>Task Approval)
- Administrators run the standard report R318 for that specific evaluation and emails it to the evaluator
How will the learner access the follow-up evaluation?
- After the classroom end date, the follow-up evaluation will be visible to the learner in three possible locations:
- Active Learning listing (To Do listing)
- Knowledge Center > Assessment section (must turn on in the Knowledge Center set up section of Module Properties), or
- Assessment Workflow widget for the Home Page
- An evaluation link can be embedded in the email template using the parameters:
- {external_evaluation_workflow_launch_link} The link can be used only once to launch the evaluation – once used, the link will no longer work and the email recipient must access from within NTS
- {evaluation_workflow_launch_link} The link requires the email recipient to log into NTS
Alternative Evaluation Models
ADDIE Model
If you use this model for designing and developing learning events, you are expected to identify categories which relate to the learning objects. As these objectives are identified, how do you want the learner to remember them?
- Create assessment questions during the development phase that can then be created in NTS.
- These questions (if gradable types) can be added to a Standard Exam type, allowing you to grade/assess the learners recollection of the information (note that it does NOT imply understanding).
- A Standard Assessment exam can be linked to the Learning Module in the Module Properties “Assign Exam/Certification/Evaluation” step.
- It can also be converted to an assignable learning module (see Import Resource) and bundled with the learning module(s) in a Program, which allows the flexibility of sequencing not permitted by the Module Properties method.
Brinkerhoff’s Success Case Method
This module focuses on not just the learning event itself, but rather than entire process. It uses inquiry to locate successes and failures to identify if the learning event requires updating or do the pre- and post- support need improvement. This method recognizes that the learning cannot be evaluated without context. As this is a more intensive analysis, it may be prudent to use it for programs with a high priority or impact.