By Julie Kwan and Stephanie Clark

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.

Adoption of a New Methodology
Premier Consultants (a pseudonym) is a consulting firm that was established in 1996 and currently employs over 4,000 employees across the United States, Europe, and Asia. In mid-2010, the company unveiled an implementation methodology, called Integrate, to all U.S. consulting employees. This new methodology was designed to support consultants on their project work and help the company achieve its goals of global growth and recognition.

Incorporation Starts with Training
To promote use of Integrate among U.S. employees, the learning and development (L&D) department designed SmartStart, a two-day training course. The purpose of the course was to educate participants on the relevance of Integrate, navigation of its company intranet site, and importance of training their teams on it. All U.S. managers through vice presidents were to attend the training course and start using Integrate on their projects beginning September 1, 2010.

Evaluation of the Course
In the fall of 2010, leaders of Premier Consultants requested an evaluation of the training course. A team of two graduate students from Boise State University conducted a summative evaluation of SmartStart to determine whether its desired outcomes have been achieved, i.e., if the project leaders are incorporating Integrate into their projects and educating their teams on how to apply Integrate to their projects. Using Scriven’s (2007) Key Evaluation Checklist as a framework, the evaluation team looked at both the processes and outcomes of the training course and analyzed two process dimensions and two outcome dimensions (see Figure 1).

Figure 1. Evaluation dimensions and questions.

A combination of surveys and interviews was used to obtain the data on these four dimensions. Anonymous web-based surveys were conducted with three groups of 894 participants (managers, senior leaders, and consultants), and about 20% of them submitted the surveys. Interviews were conducted to gather secondary data, and 20 interviewees were selected using a stratified random sample technique.

Analysis of Data
The evaluation team used the survey data as the primary source to determine the main rating for each of the four dimensions. The team then triangulated the survey results with the interview rating to determine the final rating on each dimension. Ratings were based on a four-level rubric: excellent, good, fair, and poor.

Process Dimensions
Effectiveness of Integrate advocates received an excellent rating because this role was successful in promoting buy-in of the methodology during the training course. Managerial support received a good rating because there has been an overall positive encouragement by project leaders to incorporate the Integrate methodology into projects, whether current or future.

Outcome Dimensions
Level of adoption received a good rating because there is a general attitude of support and adoption for the methodology across the levels of the company. Education of lower-level employees received a fair rating because much of the education still had not taken place. However, the evaluation team anticipates that as current projects are completed and new projects begin, both the level of adoption and the education of lower-level employees will increase.

Combining the results of the four dimensions, the evaluation team determined that the overall quality of SmartStart is good, as seen in Table 1.

Table 1. Dimensions and Weight with Overall Rating
Findings and Recommendations
The strengths of the SmartStart training course include effective leadership for promoting adoption of Integrate. During the course, Integrate advocates demonstrated strong support for the new methodology, effectively promoting buy-in from managers. Surveys and interviews showed that adoption of the methodology across the company is increasing. Given time, the evaluation team anticipates adoption and use of Integrate on projects will continue to grow.

The weaknesses of the course include the slow pace in education of lower-level employees in the use of Integrate. Level of adoption is a critical dimension of the course, and education of lower-level employees is a very important dimension of the course. Due to their respective importance weightings, the education of lower-level employees in the use of Integrate should be addressed as a top priority. As education is addressed, the level of adoption is expected to increase.

Though the training course received a rating of good, the evaluation team feels that if more time was given between the conclusion of the training course and the evaluation, the results would likely be higher. Only one and a half months had elapsed between the course and the evaluation’s data collection. Comments and feedback received during the interviews and from the surveys’ open-ended questions indicated that many respondents were working on projects for which processes and deliverables had been determined prior to the launch of Integrate.

References
Scriven, M. (2007). Key evaluation checklist. Retrieved from www.wmich.edu/evalctr/archive_checklists/kec_feb07.pdf

About the Authors

Julie Kwan is a curriculum developer for a professional services firm. She will complete her master’s degree in Instructional & Performance Technology from Boise State University in 2013. Julie may be reached at kwan_julie@yahoo.com.



Stephanie Clark is the managing editor for the curriculum lab team for an international computer training company. She will complete her master’s degree in Instructional & Performance Technology from Boise State University in 2012. Stephanie may be reached at stephaniekclark@yahoo.com.