By Laura Flanagan, Lindsay Benjegerdes, and Clinton Tyler

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.

ACME Biotech
ACME is a Biotech company in the U.S. that manufactures reagents for life sciences and research. Many of their products are sold to University labs, Hospital Clinical labs, as well as Pharmaceutical companies. The company sells over 2,000 products including kits and biological or chemical reagents. Scientists use these kits and reagents to perform biological tests or manipulate components of living cells.

Evaluation of a Training Program
In 2009, ACME instituted the Process Equipment Training (PET) program for its bottling procedure. The intended purpose was to reduce customer complaints of leaking reagent bottles through an increase in equipment operator confidence and a consistency in the bottling process (see Figure 1). An evaluation team, made up of three evaluators from Boise State University, was asked by the Engineering Manager to evaluate the training program. The primary focus of this summative evaluation was to determine if the PET program resulted in an increase of operator confidence and improved performance of operators on bottling equipment.

Figure 1. Intended outcomes of the training program.


Evaluation Dimensions

To evaluate the overall value or quality of the PET program, the team followed Scriven’s (2007) Key Evaluation Checklist as the overall evaluation framework. They began by conducting an interview with the Engineering Manager of ACME and gathering extant program data in order to determine the priorities of the program. After analyzing the data, they identified six program dimensions that would become the basis of the evaluation (see Table 1).

Next, the team had to define the importance of each dimension in relation to its impact on the program (Davidson, 2005). To determine the importance, or weight, of each dimension, the team interviewed the Engineering Manager, the Equipment Operators, and the Trainer of the PET program to gather their perspectives of the program. The results of the interviews were compiled and an importance weighting was determined for each of the dimensions. Numerical values from one to three were assigned to the levels of importance weighting (see Table 1).

Table 1. PET Evaluation Dimensions and Importance Weighting

Program Dimension Importance Weighting Reasoning
  1. Content Specifications
Extremely Important (3) Stakeholders believe the content specifications are extremely important. The setup and operation of the equipment need to be thorough. The technical accuracy is critical for proper operation of the equipment.
  1. Trainer Qualifications
Very Important (2) Stakeholders believe if the trainers are credible, that will transfer into success for the operators of the equipment.
  1. Equipment Operator Confidence
Very Important (2) Stakeholders believe the operator’s confidence in running the equipment is a critical component for efficient and productive output from the equipment.
  1. Successful Learning for Equipment Operators
Very Important (2) Stakeholders believe it is very important for the equipment operators to successfully learn how to operate the equipment for increased consistency in the bottling line.
  1. Production Line Performance
Very Important (2) Stakeholders believe a consistent bottling process will result in decreased customer complaints.
  1. Equipment Performance
Important (1) Stakeholders believe  if the equipment performance is optimal, it will lead to decreased customer complaints.

Findings and Recommendations
In order to evaluate the selected dimensions, the team interviewed the Engineering Manager, reviewed performance data from the training, conducted surveys of the Equipment Operators, and gathered Operations Equipment Effectiveness (OEE) metrics from the equipment to determine consistency in the bottling process. Each dimension was rated on a three-point scale; Poor (1), Good (2), and Excellent (3). From the data gathered, a total score was calculated by multiplying importance weighting ratings by the quality rating of each of the dimensions.

The summation of individual dimension ratings was then used to represent an overall rating of the program (see Table 2). During the data collection phase, the evaluation team realized data for equipment performance prior to training was not available, and a new method of collecting data was placed after the training event. Therefore, the equipment performance dimension was not included in the dimension rating.

Table 2. PET Evaluation Dimensions and Weighting with Overall Score

Program Dimension

Importance Weighting

Quality Rating

Score

  1. Content Specifications
Extremely Important (3) Good (2)

6

  1. Trainer Qualifications
Very Important (2) Excellent (3)

6

  1. Equipment Operator Confidence
Very Important (2) Excellent (3)

6

  1. Successful Learning for Equipment Operators
Very Important (2) Good (2)

4

  1. Production Line Performance
Very Important (2) Good (2)

4

  1. Equipment Performance*
Important (1) NA

NA

Total Score

26

*Equipment Performance dimension could not be evaluated due to a lack of pre-training data.

The evaluation team determined the overall quality of the PET Program is Good, based on the total score of 26 against the overall rubric (see Table 3).

Table 3. Overall Rubric

Excellent

Good

Poor

Score of 27-33

Score of 19-26

Score of 11-18

 

In evaluating the individual dimensions, the team found both strengths and a weakness. The strengths of the program included well-qualified trainers, increased confidence of the operators on the equipment as a result of the training, and an increase in production line performance from an estimated 40% prior to training to a calculated 53.8% after. The evaluation team considered the content specifications, successful learning for equipment operators, and pre-training as areas for further review. Improvements in these areas would bring the overall quality of the PET program to Excellent.

The findings of the evaluation were presented to the Engineering Manager and recommendations were made for continued success and improvement of the PET program. Likely, the most important recommendation is one that would impact the dimension of greatest value to the Engineering department– the content specifications. The team recommended  the Engineering department carefully aligns the Standard Operating Procedures (SOPs) and training content to reach a higher percentage of correlation. In order to achieve this, the team recommended performance skills of the equipment operators which present the greatest challenge be analyzed to reveal areas of the SOP that require change. The team also recommended the department develop and implement a skills performance test of the operators at regular intervals throughout their employment with the company.

References
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.

Scriven, M. (2007). Key evaluation checklist. Retrieved from http://www.wmich.edu/evalctr/archive_checklists/kec_feb07.pdf

Author Bios
Laura Flanagan is a Scientist and Instructional Designer at Promega Corporation in Madison, Wisconsin. She will complete her master’s degree in Instructional and Performance Technology in 2012 and may be reached at laura.flanagan@promega.com.
Lindsay Benjegerdes is an Instructional Designer at Chenega. She currently resides in the mountains of Colorado. She will finish her master’s degree in Instructional and Performance Technology in 2013.
Clint Tyler is the Appetizer Manager for a food processing company in Ontario, Oregon. He will complete his master’s degree in Instructional and Performance Technology in 2013.