By Susan Lancaster and Karen Muranaka

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Organizational Performance and Workplace Learning Department.

Company Background
Established in October 1978, Matrix Inc. (a pseudonym) is a global leader in the semiconductor industry. Matrix designs, builds, and sells memory devices that are used in consumer and business products like computers, tablets, smartphones, servers, and automobiles. The semiconductor business is similar to a rollercoaster: It has high climbs, sharp turns, and low drops. The new employees hired often reflect whether Matrix is in an upswing or a downturn. As a result, programs that support new employees have had to adapt over the years, including the company’s New Employee Orientation (NEO) programs. Currently, all new employees must complete a one-day, company-wide, NEO program that covers human resource–related topics, such as compensation, benefits, and company policies. In addition, the research and development (R&D) department, like many other departments at Matrix, provides its own orientation program.

Evaluation Purpose
The R&D Workforce Development group supports the department’s NEO program, which has been adjusted multiple times to improve efficiency and reduce costs. Because of these changes, the R&D Workforce Development manager supported an evaluation of the R&D New Employee Orientation program by a team of students from Boise State University’s Organizational Performance and Workplace Learning department. As the evaluation client, the Workforce Development manager wanted to determine the overall quality of the program in terms of whether it was meeting its desired objectives.

Training Impact Model
As a foundational piece of the evaluation, the evaluation team created a training impact model that illustrated the intended impact of the R&D NEO program, shown in Table 1. The training impact model describes the knowledge and behaviors that new employees should demonstrate as a result of the training and the job performance and organizational outcomes (Brinkerhoff, 2006, p. 71-82).

Table 1. R&D NEO Training Impact Model

Inputs or Resources Activities Program Capabilities Critical Actions Key Results Business Goals
Instructional designerVisual aids

Onboarding spreadsheet

Outputs from supporting business process, including communication between hiring manager, supervisor, NEO instructor, and new hire

Certification requirements

Supervisor

Workforce development staff

Learning management system (LMS)

Training room

Instructor

Develop presentation and activitiesGather visual aids

Create onboarding spreadsheet from SharePoint site

Set up a team to define how staff members communicate with each other (such as email, page, IM, etc.)

Area supervisors create required certifications for each job title

Workforce development creates profile in LMS

Dedicate one room for NEOs

Instructor instructs classes

Knowledge of how to properly wear cleanroom attireKnowledge of how to access information on hazardous chemicals and safety

Knowledge on how to use LMS

Knowledge of the basic production process

Knowledge of how and where to securely store documents

Knowledge of Matrix’s policies for protecting intellectual property

Knowledge of emergency contacts (phone numbers, etc.)

Dress correctly in cleanroom attireLook-up information about chemical safety and health hazards without assistance

Register for classes, take assessments for training classes, view computer-based training (CBT), view training documents, and sign off on certifications

Recognize and report security breaches

Identify internal customers to respond to area needs

Respond to non-emergency and emergency situations

Store work documents in secure locations

Area particle measurements are within control limitsEmployees produce high quality product

Employees work in a safe environment

Employees correctly perform procedure and troubleshoot issues

Matrix intellectual properties are secure

Employees are working together with their internal customers

Employees are efficient

Employees’ certifications are maintained according to departmental requirements

High quality productsIncreased market share/sales

Reduced workman compensation and other safety-related claims and lawsuits

Reductions in costs

Competitive advantage

 

Evaluation Question and Dimensions
After the client and evaluation team reviewed the training impact model, they worked together to determine the overall evaluation question and the dimensions to be investigated. The client felt that the quality of the program could be measured by determining whether the NEO curriculum was well designed and cost effective and whether the NEO participants, whose job roles, experience, and education can vary widely, were able to apply the information from NEO when they report to their work areas. Because the program participants report to different areas within the department, the orientation content also needs to target most, if not all, participants and include information that is essential for multiple areas within the department.

As a result, the evaluation of the NEO program was designed to answer this question:

How well is the R&D New Employee Orientation program doing in the areas of curriculum design, transfer of knowledge, and cost?

The evaluation team investigated one process dimension (curriculum design) and two outcome dimensions (knowledge transfer and cost). The team weighted these dimensions based on stakeholders’ knowledge and after seeing the connections in the impact model that indicated the dimensions’ importance to the success of the program, as described in Table 2. With these dimensions, the evaluation team not only assessed the merit of the program but also captured worth-based data, such as the actual costs of the program, which the client could use to drive improvements to the NEO program.

Table 2. Dimensions and Importance Weighting

Category Dimension Importance Weighting Rationale for Importance Weighting
Process 1. Curriculum Design
How well is the curriculum designed to provide relevant content and meet the objectives?
Very Important A well-designed curriculum should facilitate learning. In addition, providing irrelevant content wastes resources, time, and money.
Outcome 2. Knowledge Transfer
Are the participants able to transfer the knowledge gained from the training to the work area?
Extremely Important If the knowledge is not being transferred, these gaps need to be identified.
3. Cost
Are the costs of running the NEO program aligned with industry standards?
Very Important If the program is cost effective, it helps the organization’s profitability.

 

Evaluation Methodology
The evaluation team followed a systematic evaluation procedure based on Scriven’s (2003) key evaluation checklist, which assists evaluators to ensure that “all important ingredients that will allow valid evaluative conclusions [emphasis in the original] to be drawn are included” (Davidson, 2005, p. 5).

Good evaluation practices “require the researcher to triangulate, that is, to use multiple methods, data sources, and researchers to enhance the validity of research findings” (Mathison, 1988, p. 13). Thus, the evaluation team collected data from multiple sources using multiple data collection methods so that the data could be triangulated to generate more credible conclusions and recommendations. These data collection methods included:

  • Surveys of subject matter experts, workforce development specialists, NEO participants, and managers
  • Interviews with NEO participants
  • Reviews of extant data, such as reports, job profiles, job descriptions, training certifications, and instructional materials
  • A literature review to determine best practices and benchmark costs for new employee onboarding or orientation programs

The dimensions chosen applied to several levels of Kirkpatrick’s four-level evaluation model (2006):

  • Dimension 1: Curriculum Design (Level 1 Reaction). The team collected input from recent NEO participants about specific areas of the curriculum design and whether the participants believed the training was relevant to their actual work.
  • Dimension 2: Knowledge Transfer (Level 3 Behavior). The team collected input from supervisors and subject matter experts to determine how well new employees demonstrated the learning from NEO in the workplace. Also, in interviews with recent participants, the evaluation team asked questions that required participants to demonstrate knowledge of NEO topics.
  • Dimension 3: Cost (Level 4 Results). Kirkpatrick and Kirkpatrick (2006) define results as “the final results that occurred because the participants attended the program … and can include increased production, improved quality, decreased costs, reduced turnover, and higher profits” (p. 25). While knowing the costs of the NEO program does not directly demonstrate Level 4 results, having this data should be beneficial, if the Workforce Development manager decides to evaluate the costs of the program against the benefits it provides.

Evaluation Findings

Dimension 1: Curriculum Design. As sources of primary data, the evaluation team surveyed four job roles (recent NEO participants, their supervisors, subject matter experts, and workforce development specialists) to determine the relevance of the content and the quality of the curriculum design. As a secondary data, the team reviewed extant data from job profiles and training profiles to determine whether NEO topics were included. The dimensional quality was rated as acceptable.

Dimension 2: Knowledge Transfer. The evaluation team obtained the primary data by surveying supervisors, who evaluate participants’ job performance, and area subject matter experts, who observe participants’ work, to determine how well NEO participants demonstrate knowledge of NEO topics. The evaluation team also conducted face-to-face interviews with recent NEO participants to determine how well they were able to apply the learning in the workplace. The dimensional quality was rated as acceptable.

Dimension 3: Cost. The evaluation team calculated the overall costs of the NEO program and compared them to an industry benchmark. To determine the costs of the program, the evaluation team worked with the Employee Infodata group (pseudonym) to obtain fully loaded, hourly rates, which include salary and benefits, for the most common job roles of NEO participants. In addition, the Workforce Development specialist who administers and instructs the NEO program tracked her time for four NEO sessions and identified the roles that participated in these NEO sessions. Using the fully loaded salaries for the Workforce Development specialist and all participants, the evaluation team was thus able to calculate an average hourly cost per NEO session.

As the comparison benchmark, the evaluation team used the cost per learning hour used (consolidated) from the American Society for Training and Development (ASTD) 2013 State of the Industry. This $89 baseline represents the “cost of providing one hour of learning to an employee” (p. 23) and includes both up-front development costs and the time employees spend in learning. The average cost per participant for one hour of NEO for the time frame considered was $94. Because NEO content has been unchanged for the past few years, the evaluation team did not include development costs in its analysis of this dimension. The dimensional quality was rated as good.

Overall Evaluation Rating

With the extremely important dimension (Knowledge Transfer) being acceptable and the two very important dimensions (Curriculum Design and Cost) being acceptable and good, the R&D NEO program received a rating of acceptable. Table 3 is the quantitative synthesis rubric used to determine this overall rating.

Table 3. Evaluation Synthesis Rubric

Category Dimension Importance Weighting (IW) Raw Result IW x Raw Result = Weighted Result IW x Min Result (1) IW x Max Result (4)
Process Curriculum Design Very Important (1) Acceptable (2) 2 1 x 1=1 1 x 4=4
Outcome Knowledge Transfer Extremely Important (2) Acceptable (2) 4 2 x 1=2 2 x 4=8
Outcome Cost Very Important (1) Good (3) 3 1 x 1=1 1 x 4=4
Totals 9 4 16
Synthesis Rubric (Range between 4 and 16)
Improvement Opportunity: 4 to 7 Acceptable: 8 to 10 Good: 11 to 13 Excellent: 14 to 16

 

Strengths and Weaknesses of the NEO Program
In addition to providing an overall rating of the NEO program, the evaluation revealed a few strengths and weaknesses about the program.

Strengths

  • In interviews with participants, the information in the following sections of NEO received a score of outstanding for knowledge transfer (with at least six of seven participants demonstrating knowledge):
    • Cleanroom protocol
    • Safety/Fire evacuation
    • Matrix learning management system
  • In surveys for supervisors and subject matter experts, manufacturing software tools introduced in NEO received scores that indicated high relevance of content to the workplace (an aspect of curriculum design).
  • The cost of the program was close to an industry benchmark.

Weaknesses

  • NEO provides an overview of LM (a pseudonym) tools, but surveys for supervisors and subject matter experts indicated that not all areas use all of the tools, or the area uses different tools.
  • In interviews, none of the participants had used the Matrix document manager system, because their jobs do not require its use immediately after NEO.
  • None of the NEO participants interviewed were able to recall information on total productive management (TPM), although follow-up questions revealed that they had performed work that supported TPM.
  • Surveys for supervisors and subject matter experts indicated that the transfer of knowledge for manufacturing processes was slightly below their expectations.
  • Not all participants are being provided with training plans after NEO.

Post-Evaluation Results
The evaluation team presented the evaluation results to the client and other evaluation stakeholders. The evaluation team recommended several improvements, such as reducing the amount of information to prevent cognitive overload and introducing the use of an advance organizer that participants could reference later as a job aid. The client was anticipating a higher response rate from the subject matter experts, but was not surprised by the low number of responses. The response rate of NEO participants in both surveys and interviews was higher than expected, and the client and stakeholders agreed that the evaluation provided valuable and actionable data.

Ultimately, the results from the evaluation confirmed the client’s perceptions regarding the NEO program. In addition, having data-driven information will likely trigger improvements to the NEO program. The evaluation team is grateful for having such a great learning opportunity that also demonstrated the value that evaluations can have in the workplace.

References
American Society for Training & Development. (2013). 2013 State of the industry.

Brinkerhoff, R. O. (2006). Telling training’s story: Evaluation made simple, credible, and effective. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage Publications, Inc.

Kirkpatrick, D.L., & Kirkpatrick, J.D. (2006). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Mathison, S. (1988). Why triangulate? Educational researcher, 17 (2), 13-17.

Scriven, M. (2007). Key evaluation checklist. Retrieved from www.wmich.edu/evalctr/archive_checklists/kec_feb07.pdf

About the Authors
SusanLancaster
Susan Lancaster is an R&D workforce development specialist for a semiconductor manufacturing company. She supports the department onboarding program and teaches quality manufacturing classes. She plans to complete the Organizational Performance and Workplace Learning master’s degree program through Boise State University in December 2016. Susan may be contacted at susanl@cableone.net

 

 

KarenMuranakaKaren Muranaka has 25+ years of technical communications experience supporting information systems, manufacturing, supply chain, and human resources. She is a graduate student in the Organizational Performance and Workforce Learning program at Boise State University and will complete her master’s degree in May 2017. Karen may be contacted at karenmuranaka@u.boisestate.edu