By Susan Lancaster and Karen Muranaka
Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Organizational Performance and Workplace Learning Department.
Established in October 1978, Matrix Inc. (a pseudonym) is a global leader in the semiconductor industry. Matrix designs, builds, and sells memory devices that are used in consumer and business products like computers, tablets, smartphones, servers, and automobiles. The semiconductor business is similar to a rollercoaster: It has high climbs, sharp turns, and low drops. The new employees hired often reflect whether Matrix is in an upswing or a downturn. As a result, programs that support new employees have had to adapt over the years, including the company’s New Employee Orientation (NEO) programs. Currently, all new employees must complete a one-day, company-wide, NEO program that covers human resource–related topics, such as compensation, benefits, and company policies. In addition, the research and development (R&D) department, like many other departments at Matrix, provides its own orientation program.
The R&D Workforce Development group supports the department’s NEO program, which has been adjusted multiple times to improve efficiency and reduce costs. Because of these changes, the R&D Workforce Development manager supported an evaluation of the R&D New Employee Orientation program by a team of students from Boise State University’s Organizational Performance and Workplace Learning department. As the evaluation client, the Workforce Development manager wanted to determine the overall quality of the program in terms of whether it was meeting its desired objectives.
Training Impact Model
As a foundational piece of the evaluation, the evaluation team created a training impact model that illustrated the intended impact of the R&D NEO program, shown in Table 1. The training impact model describes the knowledge and behaviors that new employees should demonstrate as a result of the training and the job performance and organizational outcomes (Brinkerhoff, 2006, p. 71-82).
Table 1. R&D NEO Training Impact Model
|Inputs or Resources||Activities||Program Capabilities||Critical Actions||Key Results||Business Goals|
|Instructional designerVisual aids
Outputs from supporting business process, including communication between hiring manager, supervisor, NEO instructor, and new hire
Workforce development staff
Learning management system (LMS)
|Develop presentation and activitiesGather visual aids
Create onboarding spreadsheet from SharePoint site
Set up a team to define how staff members communicate with each other (such as email, page, IM, etc.)
Area supervisors create required certifications for each job title
Workforce development creates profile in LMS
Dedicate one room for NEOs
Instructor instructs classes
|Knowledge of how to properly wear cleanroom attireKnowledge of how to access information on hazardous chemicals and safety
Knowledge on how to use LMS
Knowledge of the basic production process
Knowledge of how and where to securely store documents
Knowledge of Matrix’s policies for protecting intellectual property
Knowledge of emergency contacts (phone numbers, etc.)
|Dress correctly in cleanroom attireLook-up information about chemical safety and health hazards without assistance
Register for classes, take assessments for training classes, view computer-based training (CBT), view training documents, and sign off on certifications
Recognize and report security breaches
Identify internal customers to respond to area needs
Respond to non-emergency and emergency situations
Store work documents in secure locations
|Area particle measurements are within control limitsEmployees produce high quality product
Employees work in a safe environment
Employees correctly perform procedure and troubleshoot issues
Matrix intellectual properties are secure
Employees are working together with their internal customers
Employees are efficient
Employees’ certifications are maintained according to departmental requirements
|High quality productsIncreased market share/sales
Reduced workman compensation and other safety-related claims and lawsuits
Reductions in costs
Evaluation Question and Dimensions
After the client and evaluation team reviewed the training impact model, they worked together to determine the overall evaluation question and the dimensions to be investigated. The client felt that the quality of the program could be measured by determining whether the NEO curriculum was well designed and cost effective and whether the NEO participants, whose job roles, experience, and education can vary widely, were able to apply the information from NEO when they report to their work areas. Because the program participants report to different areas within the department, the orientation content also needs to target most, if not all, participants and include information that is essential for multiple areas within the department.
As a result, the evaluation of the NEO program was designed to answer this question:
How well is the R&D New Employee Orientation program doing in the areas of curriculum design, transfer of knowledge, and cost?
The evaluation team investigated one process dimension (curriculum design) and two outcome dimensions (knowledge transfer and cost). The team weighted these dimensions based on stakeholders’ knowledge and after seeing the connections in the impact model that indicated the dimensions’ importance to the success of the program, as described in Table 2. With these dimensions, the evaluation team not only assessed the merit of the program but also captured worth-based data, such as the actual costs of the program, which the client could use to drive improvements to the NEO program.
Table 2. Dimensions and Importance Weighting
|Category||Dimension||Importance Weighting||Rationale for Importance Weighting|
|Process||1. Curriculum Design
How well is the curriculum designed to provide relevant content and meet the objectives?
|Very Important||A well-designed curriculum should facilitate learning. In addition, providing irrelevant content wastes resources, time, and money.|
|Outcome||2. Knowledge Transfer
Are the participants able to transfer the knowledge gained from the training to the work area?
|Extremely Important||If the knowledge is not being transferred, these gaps need to be identified.|
Are the costs of running the NEO program aligned with industry standards?
|Very Important||If the program is cost effective, it helps the organization’s profitability.|
The evaluation team followed a systematic evaluation procedure based on Scriven’s (2003) key evaluation checklist, which assists evaluators to ensure that “all important ingredients that will allow valid evaluative conclusions [emphasis in the original] to be drawn are included” (Davidson, 2005, p. 5).
Good evaluation practices “require the researcher to triangulate, that is, to use multiple methods, data sources, and researchers to enhance the validity of research findings” (Mathison, 1988, p. 13). Thus, the evaluation team collected data from multiple sources using multiple data collection methods so that the data could be triangulated to generate more credible conclusions and recommendations. These data collection methods included:
- Surveys of subject matter experts, workforce development specialists, NEO participants, and managers
- Interviews with NEO participants
- Reviews of extant data, such as reports, job profiles, job descriptions, training certifications, and instructional materials
- A literature review to determine best practices and benchmark costs for new employee onboarding or orientation programs
The dimensions chosen applied to several levels of Kirkpatrick’s four-level evaluation model (2006):
- Dimension 1: Curriculum Design (Level 1 Reaction). The team collected input from recent NEO participants about specific areas of the curriculum design and whether the participants believed the training was relevant to their actual work.
- Dimension 2: Knowledge Transfer (Level 3 Behavior). The team collected input from supervisors and subject matter experts to determine how well new employees demonstrated the learning from NEO in the workplace. Also, in interviews with recent participants, the evaluation team asked questions that required participants to demonstrate knowledge of NEO topics.
- Dimension 3: Cost (Level 4 Results). Kirkpatrick and Kirkpatrick (2006) define results as “the final results that occurred because the participants attended the program … and can include increased production, improved quality, decreased costs, reduced turnover, and higher profits” (p. 25). While knowing the costs of the NEO program does not directly demonstrate Level 4 results, having this data should be beneficial, if the Workforce Development manager decides to evaluate the costs of the program against the benefits it provides.
Dimension 1: Curriculum Design. As sources of primary data, the evaluation team surveyed four job roles (recent NEO participants, their supervisors, subject matter experts, and workforce development specialists) to determine the relevance of the content and the quality of the curriculum design. As a secondary data, the team reviewed extant data from job profiles and training profiles to determine whether NEO topics were included. The dimensional quality was rated as acceptable.
Dimension 2: Knowledge Transfer. The evaluation team obtained the primary data by surveying supervisors, who evaluate participants’ job performance, and area subject matter experts, who observe participants’ work, to determine how well NEO participants demonstrate knowledge of NEO topics. The evaluation team also conducted face-to-face interviews with recent NEO participants to determine how well they were able to apply the learning in the workplace. The dimensional quality was rated as acceptable.
Dimension 3: Cost. The evaluation team calculated the overall costs of the NEO program and compared them to an industry benchmark. To determine the costs of the program, the evaluation team worked with the Employee Infodata group (pseudonym) to obtain fully loaded, hourly rates, which include salary and benefits, for the most common job roles of NEO participants. In addition, the Workforce Development specialist who administers and instructs the NEO program tracked her time for four NEO sessions and identified the roles that participated in these NEO sessions. Using the fully loaded salaries for the Workforce Development specialist and all participants, the evaluation team was thus able to calculate an average hourly cost per NEO session.
As the comparison benchmark, the evaluation team used the cost per learning hour used (consolidated) from the American Society for Training and Development (ASTD) 2013 State of the Industry. This $89 baseline represents the “cost of providing one hour of learning to an employee” (p. 23) and includes both up-front development costs and the time employees spend in learning. The average cost per participant for one hour of NEO for the time frame considered was $94. Because NEO content has been unchanged for the past few years, the evaluation team did not include development costs in its analysis of this dimension. The dimensional quality was rated as good.
Overall Evaluation Rating
With the extremely important dimension (Knowledge Transfer) being acceptable and the two very important dimensions (Curriculum Design and Cost) being acceptable and good, the R&D NEO program received a rating of acceptable. Table 3 is the quantitative synthesis rubric used to determine this overall rating.
Table 3. Evaluation Synthesis Rubric
|Category||Dimension||Importance Weighting (IW)||Raw Result||IW x Raw Result = Weighted Result||IW x Min Result (1)||IW x Max Result (4)|
|Process||Curriculum Design||Very Important (1)||Acceptable (2)||2||1 x 1=1||1 x 4=4|
|Outcome||Knowledge Transfer||Extremely Important (2)||Acceptable (2)||4||2 x 1=2||2 x 4=8|
|Outcome||Cost||Very Important (1)||Good (3)||3||1 x 1=1||1 x 4=4|
|Synthesis Rubric (Range between 4 and 16)|
|Improvement Opportunity: 4 to 7||Acceptable: 8 to 10||Good: 11 to 13||Excellent: 14 to 16|
Strengths and Weaknesses of the NEO Program
In addition to providing an overall rating of the NEO program, the evaluation revealed a few strengths and weaknesses about the program.
- In interviews with participants, the information in the following sections of NEO received a score of outstanding for knowledge transfer (with at least six of seven participants demonstrating knowledge):
- Cleanroom protocol
- Safety/Fire evacuation
- Matrix learning management system
- In surveys for supervisors and subject matter experts, manufacturing software tools introduced in NEO received scores that indicated high relevance of content to the workplace (an aspect of curriculum design).
- The cost of the program was close to an industry benchmark.
- NEO provides an overview of LM (a pseudonym) tools, but surveys for supervisors and subject matter experts indicated that not all areas use all of the tools, or the area uses different tools.
- In interviews, none of the participants had used the Matrix document manager system, because their jobs do not require its use immediately after NEO.
- None of the NEO participants interviewed were able to recall information on total productive management (TPM), although follow-up questions revealed that they had performed work that supported TPM.
- Surveys for supervisors and subject matter experts indicated that the transfer of knowledge for manufacturing processes was slightly below their expectations.
- Not all participants are being provided with training plans after NEO.
The evaluation team presented the evaluation results to the client and other evaluation stakeholders. The evaluation team recommended several improvements, such as reducing the amount of information to prevent cognitive overload and introducing the use of an advance organizer that participants could reference later as a job aid. The client was anticipating a higher response rate from the subject matter experts, but was not surprised by the low number of responses. The response rate of NEO participants in both surveys and interviews was higher than expected, and the client and stakeholders agreed that the evaluation provided valuable and actionable data.
Ultimately, the results from the evaluation confirmed the client’s perceptions regarding the NEO program. In addition, having data-driven information will likely trigger improvements to the NEO program. The evaluation team is grateful for having such a great learning opportunity that also demonstrated the value that evaluations can have in the workplace.
American Society for Training & Development. (2013). 2013 State of the industry.
Brinkerhoff, R. O. (2006). Telling training’s story: Evaluation made simple, credible, and effective. San Francisco, CA: Berrett-Koehler Publishers, Inc.
Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage Publications, Inc.
Kirkpatrick, D.L., & Kirkpatrick, J.D. (2006). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler Publishers, Inc.
Mathison, S. (1988). Why triangulate? Educational researcher, 17 (2), 13-17.
Scriven, M. (2007). Key evaluation checklist. Retrieved from www.wmich.edu/evalctr/archive_checklists/kec_feb07.pdf
About the Authors
Susan Lancaster is an R&D workforce development specialist for a semiconductor manufacturing company. She supports the department onboarding program and teaches quality manufacturing classes. She plans to complete the Organizational Performance and Workplace Learning master’s degree program through Boise State University in December 2016. Susan may be contacted at firstname.lastname@example.org
Karen Muranaka has 25+ years of technical communications experience supporting information systems, manufacturing, supply chain, and human resources. She is a graduate student in the Organizational Performance and Workforce Learning program at Boise State University and will complete her master’s degree in May 2017. Karen may be contacted at email@example.com