By Natalie Peeterse, Ann Yandell, and Larissa Catcott
Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.
Best Technical Consulting
Best Technical Consulting (BTC), an information technology (IT) consulting firm based in Missoula, Montana, has been providing help desk support to its clients since the company was founded in 2002. In January 2011, BTC began outsourcing a portion of client help desk inquiries to streamline operations. To date, the help desk functions as an integral part of a full complement of IT services offered by the company, providing superior and timely support and expertise to clients dealing with a wide variety of technical problems.
Evaluating the Help Desk Program
In August 2011, the BTC president and chief consultant requested a team of graduate students at Boise State University (authors) to conduct an evaluation of the BTC help desk program to identify its overall merit and worth to the company. The team conducted the evaluation from October to December of 2011 to determine if the BTC help desk program was meeting the needs of clients and if the program was worth what it costs in terms of time, money, and other resources.
The evaluation was considered both summative and goal based, given the extensive knowledge of program goals in existence, time constraints, and the client’s request to focus on outcomes. However, careful consideration was given to the process and costs as well as to outcomes, particularly regarding program implementation and relationships. Scriven’s Key Evaluation Checklist (KEC; 2007) was referenced to ensure the systematic application of evaluation practices and to avoid the limitations inherent to a “black box” study (Fitzpatrick, Christie, & Mark, 2009, p. 20).
The evaluation team used a BTC program logic model in conjunction with the KEC as a framework to evaluate the help desk program. Data from the process, outcomes, and cost were analyzed in an iterative fashion, not only for use in determining the absolute merit of the program, but also as key qualitative and quantitative input toward determining return on investment and overall program worth, and for inferring causation. This iterative approach provided a clearer conceptualization of the entire program and thus augmented the synthesis of data into a valid conclusion.
In consultation with the client, the evaluation team determined dimensions that reflect the most significant components of help desk activities. The team gave careful consideration to several sources in determining the dimensions of merit for this evaluation: the general “nature of the evaluand” (Scriven, 2007, p.6), program goals, the program logic model, and the values identified in the Key Evaluation Checklist (Scriven, 2007). With these sources in mind and in consultation with the client, the following five primary criteria of merit were identified:
- Implementation–Are BTC help desk requests ticketed, prioritized, and resolved in an effective manner?
- Criticality–How important is the BTC help desk to the operational success of clients?
- Client Satisfaction–Are clients satisfied with help desk support?
- Operations Alignment–Does the BTC help desk program generate time savings for BTC management?
- Return on Investment–What is the overall program return on investment?
During the course of the evaluation, the team collected and analyzed both qualitative and quantitative data from multiple sources through reviews of BTC business records, and surveys and interviews with the client employees, the outsourcing technicians, and the BTC chief consultant to come to evaluative conclusions that were valid and evidentiary. All data sources were considered to have equal merit, as requested by the client.
In synthesizing results from these five dimensions, the evaluation team found the overall quality and the overall worth of the BTC help desk program to be “Good” when measured on a four-level scale (Excellent, Good, Marginal, and Poor). Dimensional results with relative importance weighting (Very Important vs. Extremely Important) are shown in Table1.
Table 1. BTC Help Desk Program Dimensions with Overall Quality and Value Rating
|Outcome||Client Satisfaction||Extremely Important||x|
|Operations Alignment||Extremely Important||x|
|Cost||Return On Investment||Very Important||x|
 Results of Process Evaluation
Examination of help desk processes revealed that the current system was generally efficient, met or exceeded standard practice, and contributed substantially to client success. However, support processes were sometimes hindered by miscommunication that resulted in support delay and additional workload.
 Results of Outcome Evaluation
An analysis of program outcomes showed that the majority of clients were very satisfied with the services provided by BTC help desk support and greatly valued the professional expertise of the BTC chief consultant. Nevertheless, limitations in the service plan agreement contributed to disproportional workload levels and organizational goal misalignment.
 Results of Cost Evaluation
The cost evaluation involved an examination of BTC’s financial records to conduct a return on investment analysis. While the company’s gains did not make up for the company’s costs in 2009 (the first year when the help desk program was initiated), the return on investment analysis revealed substantial gains for fiscal years 2010 and 2011, and projections for 2012 indicated a positive return on investment as well. The cumulative return on investment for the four years the help desk has been in operation is 16.76% (see Table 2). This takes into account all of the costs and gains from 2009 and projected through 2012. Even with the fluctuation in the annual return on investment, the BTC help desk program is currently generating a positive return on investment. Because projections show that this trend should continue, the evaluation team determined the program to be worthwhile for the client in the foreseeable future.
Table 2. Best Technical Consulting Return on Investment
|BTC’s Return on Investment – Help Desk Program|
|Year||Annual Business Costs||Cumulative Business Costs||Annual Financial Gains||Cumulative Gains||Annual ROI|
 Strengths and Weaknesses of the Program
The evaluation team also concluded the following as the overall strengths and weaknesses of the help desk program:
- Clients trust the BTC chief consultant and value his professional expertise.
- The help desk contributes significantly to the success of client business operations.
- The outsourcing technicians are professional and knowledgeable in performing client support.
- The BTC help desk return on investment has been positive in recent years, contributing substantially to profits.
- Limitations in the service plan agreement result in a disproportional amount of workload being placed on the BTC chief consultant.
- Help desk support processes are sometimes hindered with preventable miscommunications that can result in support delays.
- Familiarity with clients contributes to excessive workload for BTC chief consultant and weakens the implementation processes.
Fitzpatrick, J., Christie, C., & Mark, M. M. (2009). Evaluation in Action: Interviews with Expert Evaluators. Thousand Oaks, CA: Sage. Click here for the full text.
Scriven, M. (2007). The Key Evaluation Checklist. Retrieved from www.wmich.edu/evalctr/ archive_checklists/kec_feb07.pdf
Natalie Peeterse works in online higher education and lives in Missoula, MT. She will complete her Master of Science degree in Instructional and Performance Technology in 2013 and may be reached at firstname.lastname@example.org.
Larisa Catcott works for a large food manufacturing company and lives in northeast Ohio. She will complete her Master of Science degree in Instructional and Performance Technology in 2013. Larisa may be reached at email@example.com.
Ann Yandell (No image available) has several years’ experience as an educator. She will complete her Master of Science in Instructional Technology and Learning in July 2012, and a graduate certificate in Human Performance Technology in December 2012. Ann may be reached at firstname.lastname@example.org.