Project Proposal Evaluation and Analytics in Instructional
E-Comm: Director Development Program Evaluation Plan
EDIT 7350: Evaluation and Analytics
Dr. Lauren Bagdy
Presented by:
Paula Borras-Patino, Stephanie Denny, Michaela Hyland, Julie Wyatt
like an evaluation of their Director Development Program, which has been created to provide a way for current directors across multiple functional areas to grow within the company specifically towards the leadership opportunity as a Vice President. The Director Development Program participants can come from a diverse set of internal areas such as merchandising, creative, technology, marketing, production, editorial, and people management The current training is online, which was adapted from an in-person training, but changed to online due to the pandemic, however, the company would like to continue the online version.
The training program is a series of seven learning modules and each module has a guiding question to help the learner gain insight into the lesson. The first module is an introduction that provides an overview of the expectations of the Director Development Program. Moving into the lessons for the training, for module one, “Self-Leadership”, the focus is on delegation and resilience Delegation is important for the Directors to help employee retention; delegating responsibility enables the employee to feel more valued with more responsibility. Resilience is discussed in terms of self-awareness in times of stress. For module two, “Leading with a Coaching Mindset”, the focus is on coaching and the various models of how to be a coach. The learners will be introduced to different coaching styles and identify which one is right for them, or the situation to use different coaching models. For module three, “Coaching Skills for Leaders”, the focus moves beyond the different coaching techniques and to how to provide feedback to employees that the learner is coaching, setting goals for their employees, and finding solutions to an employee’s job-related problems For module four, “Leading High Performing Teams”, the focus is on how to build a team and why teamwork is better than individual work in solving problems However, there can be pitfalls to teamwork, the learners discuss what causes team dysfunction and what they can do to help improve it for the better. One way of doing so is by following the “ingredients of a high-performing team” equation. For module five, “Leading your Function”, the focus is discussing why each functional area of the company is important and how the learner can use influence in their role without relying on just their position of authority. The final module is a celebration of all the learning modules as well as “Building your Leadership Brand”, the focus is exactly what the subject implies, why a leadership brand matters, and how it can help the learner in their interest in becoming a Vice President This final session will encourage self-reflection about the learner’s time in the training program as well as what the learner’s future goals are
The structure of this evaluation is summative to define the goals and objectives of the Director Development Program and develop evaluation tools to see if the goals and objectives are met. The goals and objectives are further established in the evaluation purpose further in this report. The program has built-in mechanisms for formative evaluation, though the findings of this evaluation might also provide insight for improved instruction.
Page 3
The Evaluation Team consists of the following people:
Paula Borras-Patino
Bachelor’s in Financial Engineering
M. Ed. student in Learning, Design, and Technology
Training Specialist at The Coca-Cola Company
Email: mbl13896@uga.edu
Stephanie Denny
Bachelor’s in Communication and English
MFA in Creative Writing
M.Ed. student in Learning, Design, and Technology
Higher education professor for 20 years
Email: Stephanie.Denny@uga.edu
Michaela Hyland
Bachelor’s in Economics
M.Ed. student in Learning, Design, and Technology
Instructor of high school mathematics for 8 years
Email: mchyland@uga.edu
Julie Wyatt
Bachelor’s in History Education
Master’s in Public Administration
M.Ed. student in Learning, Design, and Technology
Higher education administrator for 10 year
Email: Julie.Wyatt@uga.edu
Page 4
Evaluation Purpose
The purpose of the evaluation of the training instituted by E-Com, a company focused on clothing and lifestyle brands, is to determine whether it helps Director-level employees with time management, team-building, influencing the brand, and collaborating across functions.
Each module currently asks employees for feedback at the end by taking a survey Having these results would allow us to understand the participants’ reactions, a level on the Kirkpatrick model of evaluation represented in Figure 1, and therefore whether they find the training enjoyable and useful, even as a first impression. Requesting feedback from the Client Success Manager leading the training will also produce useful feedback about the pace of the training and common questions learners had. Therefore, an interview will be arranged for this purpose.
March 28, 2021.
Additional feedback can be obtained through observations of directors working with employees both during and after training. This will provide evidence of demonstrating learning and applied behaviors (also levels on the Kirkpatrick model) based on the topics covered in the training. For instance, how are they working through conflict? How are they assisting their team members in reaching their goals, or what is standing in their way? Asking questions like these will provide insights on possible areas to strengthen training or to add more focus.
A summative evaluation that focuses on whether the Director-level employees were successful in collaborating, delegating tasks, and coaching would provide data in these areas. Surveying the directors and the teams that they work with, both in the early stages of the training and after the training has ended, would be useful and provide data to compare
Because module one mentions an issue with lack of development opportunities at E-Com, one way to consider whether change, in the results area of the Kirkpatrick model, has been made is to review whether any Directors left the company shortly after training, and if so, what the reason is as indicated in the Exit Survey This information can be compared with previous data, before the training began. This would be one final summative evaluation.
5
Figure 1: From “Kirkpatrick Evaluation Model-combined 2,” by Rosie Williams, Brightstone,
Perspective of
Director-level employees)
Director-level employees (learners)
Primary Primary
Impact of training
Effectiveness of learning
Senior Leaders Primary Leadership Program planning
Effectiveness
Development Firm (Instructional Designer)
Client Success Manager (form development firm)
Primary Implementer
Primary Implementer Program Planning Learner oversight
Effectiveness Impact of training
Logic Model
Context
Impact of training Effectiveness
Recipient of evaluat
data collection
Recipient of evaluation
Participating in the
Recipient of evaluation
Effectivness
Involved in evaluation planning
Page 6
Involved in the evaluation planning
Recipient of evaluation
Participating in the data collection
E-Comm is an online based company with a diverse set of functional area directors going through the same leadership training program. One factor that may influence
the training program is that there may only be so many job openings available for the Vice President level, even with the training program in place. This can affect the training program participants to be more competitive and less collaborative during their learning modules.
Another factor is that the facilitator is a client success manager from the leadership development firm who designed the training, and while they can lead the discussion and follow the framework, it might not be as effective as it could be if the facilitator was an internal Vice President who can share their experience. On the other hand, if the facilitator is an internal employee, it might be seen as biased toward one functional area unless there was a rotation of facilitators, which would then lose some continuity for the training.
Population and Needs
The program is providing professional development skills, on a Director level, to work on time management, team-building, personal influence, and collaboration across functional areas. Ultimately, this training will help introduce, develop, and sharpen interpersonal skills of a director to prepare them for a promotion.
Age and Phase
An in person training version existed prior to 2020, the evaluation team does not know how long this training was in place. However, since 2020, the training has been online. Therefore the age of the training is three years old. The training is well established, but might benefit from some redesign, especially if the transition from in person to online was quickly developed or translated. Page 7
Table 2. A Simplified Logic Model in Table Form
OutputsShort-Term Outcomes Long-Term Outcomes
Resources/Inputs Activities
PowerPoint Slides
Facilitator Guides
Facilitator
Time:
Group Meetings: once a month for 3 hours
approx. every 3 to 4 weeks
Individual Coaching Sessions: once a month for approx. 1 hour before the next group meeting
Readings and Reflections: 10 to 30 minutes once a month before the group meeting
Workbook
Directors
Zoom or other platform facilitator uses Workbook
Directors
Zoom or other platform facilitator uses Peer collaboration Reflection
Instruction in 5 module and a closing 5 Group discussions
Fireside Chats with Senior Leaders
1:1 Coaching sessions
Change Initiative
Pitch Presentation Manager Check-ins
Leadership brands (VP’s identify What you value and how you are seen by others)
Continuing action plans
More confident directors
More confident employees Employees who are more closely connected
Employees with more knowledge about how to coach others
Collaboration across functional areas
Promotion of leadership practices across the company
Fewer
employees who leave because of lack of development
opportunities Higher employee retention due to the professional development opportunities available internally
Availability of a skilled talent pool for future VP leadership roles
Page 8 5 lessons Reflection journals
Participants identifying their individual coaching style
Discussion- Logic Model
There are many different resources and inputs that go into the E-Com training program. The instructional support resources include the PowerPoint slides, facilitator guide, learner workbook. Another resource is the leadership development firm, who served as the instructional designer. The client success manager from the firm who serves as the training facilitator is also an important resource. From the E-Com side, the senior leaders and managers of the director-level employees are also included as resources and inputs.
Activities and Outputs
The activities and outputs for this training include peer learning, independent study, group discussions, and reflections. There are also “fireside chats” embedded in the training, where learners will meet with senior leaders. Learners will have 1:1 coaching sessions, manager check-ins, and will participate in a change initiative pitch presentation. The training consists of six training modules and each learner will complete activities in a workbook for each module.
Outcomes
The main outcome for the E-Com leadership development program is to prepare current director-level employees to be ready to take on the role of Vice President. In the short term, they are looking to instill more confidence in the directors and to build stronger connections between current employees. The coaching sessions and instructional modules are also supposed to equip learners with more knowledge about how to coach others.
There are also long-term outcomes for the training program, including the promotion of leadership practices across the company and to increase employee retention due to the professional development opportunities available internally. This training was developed in response to former employees noting the lack of leadership development within the company, so another long-term outcome is to have fewer employees leave the company because of a lack of development opportunities.
9
Evaluation Questions
Table 3. Focus Areas, Questions, Design, and Rationale of Your Evaluation
StakeholderStakehol
der Needs
Time to meet with peers to complete lesson
Director-level employees (learners)
Evaluation Focus Areas Training effectiveness Amount of time provided for
Leadership program preparing director level employees for a VP role Cost effectiveness of the program
Senior Leaders
Time to complete reflection
Relevance to job
Time to prepare for fireside chats Prepared VP force for supporting roles and delegation training activities
3-5 Evaluation Questions
What components were most useful and least useful? What module, if
What organizational factors (eg staff capacity, leadership, resources) might have affected the effort’s implementation and outcomes? Was the program successful in developing the new VP Force? Was the effort cost effective?
Evaluation Design
Evaluation Design
Rationale
End-of module surveys In class observations
general their perspective of the success of the Training program, including their perspective about cost effectiveness of the program
Director-level employees)
Feedback from learners as well as the trainer will provide
Communicati on from other stakeholders
Success in program implementati on
Leadership Development Firm (Instructional Designer)
Prepared VP force for supporting roles and delegation Program achieving desired outcomes
Open-ended
questionnair es insights about learner needs both during and after the training?
Page 10 any, do facilitators need to spend more or less time covering? Did the workbook help reinforce the learning from the modules?
Feedback from Senior Leaders will gather their insight on how prepared the new VP’s are and in
Vice PResidents: Managers (of
Program achieving desired outcomes In class observations will provide information to determine if the program is being delivered the way it was planned, as well as if it has been successful.
Feedback from managers will provide insights about the effectiveness of the training programs.
Page 11
What components of the leadership program were successfully implemented? What topic, if any, did the learners struggle with? With the redesigned program (in person to virtual), were the same outcomes achieved?
Did the training
contribute to the desired intermediate outcomes? Are there any aspects of the VP duties and VP characteristics that were omitted in the training program and should be included in future enhanced training programs? Is it helpful that participants share their Continuing Action Plans? And do you think managers should have an input into the development of the participants CAP?
In-class observations
Feedback will be collected through interviews
firm)
Client success manager (from development
Evaluation Design
Participants that are ready to take the next developmenta l stage Commitment from the organization and the participants to respect the training schedule Support from the Management and Senior leaders communicatin g the expectations from the directors participating in the VP training programs as well as supporting the directors with the time and space needed to complete all training activities. Program achieving desired outcomes Support provided by client Feedback from the client success managers will provide insights about possible needs the program has that can help
improve the delivery of the program
Were the selected participants ready for the VP Training Program? Based on the facilitator's experience, what qualities do successful leaders typically have during training? What organizational factors (e g staff capacity, leadership, resources) might have affected the effort’s implementation and outcomes?
Feedback will be collected through interview(s) with the client success manager(s) leading the training
Page 12
Given that the program has been operational for three years and is presently in the maintenance phase, we will employ a mixed-methods approach in our evaluations. This approach will include the utilization of open-ended questionnaires to explore the topic in-depth, alongside interviews and in-class observations. Such a diverse methodology allows us to draw from a wide range of sources, including current directors actively involved in the program, current Vice Presidents who were
program participants, and former employees who previously participated in the program but have since departed from E-Com.
To mitigate the risk of sample bias and response bias, data will be collected in small groups. Sample bias may occur if some participants provide incomplete information, but by maintaining smaller groups, skilled interviewers can ensure comprehensive data collection, thus reducing bias. Similarly, response bias may arise if participants misunderstand questions or choose not to provide honest responses, potentially to protect the project. Therefore, participants will be oriented to the purpose of the evaluation, which is to maintain and enhance the training program. This emphasis on the use of gathered information to improve the current program is intended to encourage participants to be more candid and provide valuable insights.
Data Collection and Analysis
Table 4. Evaluation Data Collection and Analysis Plan
1. What components of the training, (e.g. time management, team building, influencing the brand, or collaborating across functions) were most useful and least useful?
2. Was the program successful in developing the new VP Force?
End-of-module surveys
In class observations
Open-ended questionnaires
Existing
Data: End of-Module Surveys for each module
Observation Checklist Program participants
Current Vice Presidents Page 14
Review each module presentation Obtain Permission to Access End-of Module Surveys Create observation checklist
Develop open ended questionnaire Analysis of Surveys and Observations Survey Descriptive Statistics and Thematic Analysis
Data Collection and Analysis Plan
3. Are there any aspects of the VP duties and VP characteristics that were omitted in the training program and should be included in future enhanced training programs?
Interviews
Current Vice
Review Exit Survey from employees who have left the company
Create Interview questions
Thematic Analysis
Data Collection Instruments Evaluation Questions
Question 1
What components of the training, (e.g. time management, team building, influencing the brand, or collaborating across functions) were most useful and least useful?
The end-of-module survey will be used to measure what components of the training were more useful and least useful Data Collection Method Precautions
Data Collection Tool 1 from the point of view of the trainees. The observation will try to capture any other components that the trainees may have dismissed
Participants will need to be confident that their anonymity will be respected. When it is announced that the class is being observed, the evaluator will ensure that participants are informed that their individual information will not be collected as part of the observation. To limit bias on the part of the evaluator, questions on the observation form are very specific. The evaluation team will also have more than one evaluator complete observations.
To collect information in response to question one, training sessions will be observed. The evaluator will respond using a Likert scale, which will allow for comprehensive data analysis of the range, mean, median, and mode of responses. Each module should be observed so that any gaps might be identified. For example, Module 4 might receive high marks for in-depth coverage, but Module 1 might not. Page 16
Tool 1:
Leadership Development Program Observation Checklist The purpose of this checklist is to determine the effectiveness of E-Com Group’s
Leadership Development Program modules, which focus on self-leadership, the power of coaching, coaching skills for leaders, leading high-performing teams, and leading your function.
Module Title:
Evaluator:
Date:
Instructions: In the table, please use an X to indicate the answer that most closely reflects your observations of the module. An area at the end will ask you to expand on one answer.
Strongly
Observation
1. The warm-up activity was purposeful and relevant to module objectives.
2. The instructor effectively oriented participants by reviewing the timeline.
3. An overview of the agenda was provided.
AgreeAgree DisagreeStrongly
DisagreeN/A
Observation
4. Participant interactions reflected engagement with the material.
5. The balance between discussion and lecture was appropriate.
6. Material covered in the training clearly related to the job duties of a leadership position (i.e., Vice President).
7. The training covered the material in an appropriate manner.
8. Breakout room activities supported the module objectives.
Strongly
AgreeAgree DisagreeStrongly
DisagreeN/A
Observation
9. Questions were
answered effectively by the facilitator
10. The length of this training session was appropriate.*
*Please explain your answer to Question 10 above, about the length of the training session. If possible, be specific about which objective took longer or shorter than indicated (or than necessary).
Strongly
AgreeAgree DisagreeStrongly
DisagreeN/A Page 19
Question 2
Was the program successful in developing the new VP Force?
Data Collection Method Vice-presidents will complete an open-ended questionnaire.
Precautions
The participants may or may not have reached the VP level by the time of the interview.
Data Collection Tool Description
Participants may not want to be candid and provide valuable insight; therefore, orient the participants to the purpose of the evaluation, which is to maintain and enhance the training program.
This tool will be used to gather information for analyzing whether the program is meeting its objectives. In instances where objectives are not being met, recommendations for program improvement will be formulated.
Tool 2: Leadership Development Program Questionnaire
Instructions: Ideally this evaluation will be done by a skilled interviewer in small groups of VPs, to encourage candid and
Interviewer:
Date:
Group Size:
Group #:
Question 1
Looking back over the past three years, do you perceive that the current directors can easily transition to the Vice-president level? Why or why not? Comments:
Question 2
Can current trained directors adapt their natural leadership styles to suit the various circumstances that arise at the Vice-president level? Explain.
Comments:
Question 3
When considering the current VP teams, do team members feel a sense of belonging? Explain.
Comments:
Question 4
How do you perceive the level of interest among current directors in joining the training program, and what factors might contribute to its popularity or lack thereof?
Comments:
Question 5
For directors who have undergone the coaching program, do they focus on the most important tasks and effectively delegate to their team members? Explain. Comments:
Question 6
In what ways do members of the director team respond to new tasks, and what factors might contribute to their feelings of empowerment or being overwhelmed?
Comments:
Question 7
Do directors apply the skills learned in the program when coaching their teams and assisting team members in setting goals? Explain.
Comments:
22
Question 8
Do current VPs who were promoted after completing the training program demonstrate care for the well-being, development, and productivity of their individual team members? If yes, please give an example.
Comments:
Question 9
Can you identify evidence showing that directors who completed the training program now expand their influence beyond their teams, building a strong network within E-Come Group?
Comments:
Question 10
Would you say that teams led by a VP who completed the training program share the same purpose and direction? What indicators could signify this alignment or lack thereof?
Comments:
Question 3
Are there any aspects of the VP duties and VP characteristics that were omitted in the training program and should be included in future enhanced training programs?
Feedback will be collected through interview(s) with the current
Data Collection Tool Description
Vice-Presidents and with Directors participating in the Director Development Program. The interviews will be one-on-one and will last no longer than 30 minutes.
Evaluators will need to ensure that the interviews stay confidential because the interviews could contain sensitive material about the participants. The current Vice-Presidents might not have attended this training previously or might not be aware of what the training covers.
This tool will be used in response to question three and the interview protocol will be implemented. The interviews will take place between the evaluation team and the current VPs to determine success of the program. After the conclusion of the interviews, there will be a thematic analysis based on the responses. Page 24
Tool
3:
Interview Protocol for the E-Comm: Director Development Program
Name of Interviewer
Name of Interviewee
1. Do the learning modules (time management, team building, influencing the brand, or collaborating across functions, or other) accurately reflect what the future Vice President needs to
2. In what component have the current Directors/newly promoted Vice Presidents needed the most support (time management, team building, influencing the brand, or collaborating across functions, or other)?
Comments
3. Have you seen an improvement in preparedness for the Directors/newly promoted Vice Presidents that have taken the course when compared to those who have not? Explain why or why not. Comments
4. What is the biggest strength that has come from incorporating the training program? Comments
5. What is the biggest weakness that has come from incorporating the training program? Comments
6. Is there anything that you think needs to be added (or taken away) from the training program? Comments
