By Stephanie Clark, Amanda Collins, Julie Kwan, and Allison Sesnon

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.

Introduction
In the fall of 2011, the Lean on Me Charity (LoMC; a pseudonym to protect confidentiality) chapter of Idaho (LoMCId) directors contacted Boise State University’s (BSU) Instructional & Performance Technology graduate program. They requested a team of instructional design (ID) students to develop training to help the LoMC team successfully implement their newly created Service Standards. By the end of the fall, the BSU ID team had developed a total solution that addressed not only training but all areas of the performance gap. This article shares the process that the BSU ID team followed, the resultant performance solution, and results obtained to date.

Background
LoMC is a global network of chapters operating in 52 countries. LoMC began in the early 1970s in the Eastern United States to provide temporary accommodations and support to families in need during extremely stressful situations. Today, there are more than 300 LoMC chapters around the world. The LoMCId opened in 1988. Currently, it employs 13 paid staff members and approximately 30 volunteers.

In 2010, the LoMC global office requested that all LoMCs ensure that their staff and volunteers engage in appropriate conduct while on the premises to provide excellent service to their clientele. In 2011, with input from staff and volunteers, LoMCId created Service Standards describing appropriate conduct in interactions with clientele and with one another. The Service Standards consist of a series of behavioral statements, grouped into four categories:

  • Professional
  • Example: “We respond to questions and/or concerns in a prompt and courteous manner.”
  • Respectful
  • Example: “We protect our families’ need to rest, relax, and sleep at all hours of the day.”
  • Compassionate
  • Example: “We support and encourage guests, recognize when someone is in need of help, and assist appropriately.”
  • Helpful
  • Example: “We are proactive and anticipate families’ needs.”

LoMCId wanted their ID team to provide training on these formal standards in ways that would make them part of the organization culture and improve overall performance.

Data
The BSU ID team collected both extant and other data to conduct a Performance and Cause Analysis (PCA). These initial efforts informed a subsequent Learner Analysis (LA) and a Task Analysis (TA). Table 1 lists the data sources and where the BSU ID team used them.

Table 1. Sources of Data.

Source Use
Extant 2010 LoMCId Clientele Satisfaction Survey PCA
2010 Volunteer Satisfaction Survey PCA
Personnel/volunteer records LA
Team-Generated 2011 Employee/Volunteer Questionnaire PCA, LA
Interviews with:

  • Director of chapter operations
  • Executive director
  • Volunteer coordinator
PCA, LA, TA
Focus group of three subject matter experts (SMEs) TA
Observations of LoMCId team on LoMCId-identified days TA
Interviews with other LoMC chapters across the nation TA


Performance and Cause Analysis
Based on their analysis of the performance analysis data, the BSU ID team identified the following gap and its causes.

Performance Gap
Desired performance: Staff and volunteers will apply Service Standard behaviors 90% of the time while in, or around, the LoMCId.

Current performance: Currently, staff and volunteers apply Service Standard behaviors 60% to 75% of the time while in, or around, the LoMCId.

The Service Standards tie directly to the Statement of Values for the LoMCId (the statement information is not provided in this article to protect confidentiality). By closing the performance gap, LoMCId team members will meet this statement by consistently treating their clientele with respect, support, and responsiveness.

Causes
Having identified a performance gap worth closing, the team conducted a cause analysis. Table 2 lists causes of the performance gap and the individuals who are responsible for addressing them.

Table 2. Performance Gap Causes and Solutions.

    Cause and Solution Responsibility
Environmental Causes Data There was a lack of communication to the LoMCId team on performance expectations. For instance, the LoMCId management expected their team to provide peer-to-peer feedback to help improve performance on the Service Standards. However, the SMEs felt very strongly that they did not feel it was their place to provide feedback to other volunteers. There was no formal mechanism to provide feedback about the extent to which individuals met the service standards. LoMCId directors and volunteer coordinator
There were no formal evaluations of the volunteers. Evaluations would provide areas for improvement on implementation of the Service Standards. A formal evaluation process would also help the LoMCId directors and volunteer coordinator to effectively measure progress toward demonstrating the Service Standards 90% of the time. LoMCId volunteer coordinator
Resources Some of the processes at LoMCId did not support compassionate performance by the staff. For instance, the current timing and process of explaining LoMCId rules to the clientele do not support compassionate consideration of the clientele or encourage compliance by the clientele. LoMCId directors and volunteer coordinator
Individual Causes Knowledge There was a lack of knowledge and skills related to the application of the Service Standards in the stressful situations that arise at LoMCId. Training could provide the LoMCId team a safe environment in which to “walk in each other’s shoes” and to learn how to better apply customer service skills. BSU ID team, LoMCId directors and volunteer coordinator
Capacity There were no placement criteria for volunteers to ensure that the right people with the right skills were in the right positions. LoMCId directors and volunteer coordinator

 

The challenge became how to address knowledge and skills while providing a venue to simultaneously address other environmental and individual causes. After all, if the BSU ID team addressed individual sources of the gap while ignoring the environmental ones, they would leave the client facing the same performance gap that motivated their request for training. The BSU ID team opted for a “training-plus” solution that would use the training to address skills while creating a forum to address environmental and capacity causes of the performance gap.

Learner Analysis
The learners consisted of two groups: volunteers and paid employees. The volunteer population (30) had an average age of 65. The paid employee population (13) had an average age of 38. There has been little turnover in volunteer population in the last 5 years. Table 3 describes characteristics of the LoMCId team and their impact on the training solution.

Table 3. Learner Analysis Findings.

  Learner Analysis Findings Impact on Training Solution
Attitude toward Training Many volunteers were resistant to change. The training needed to actively engage the learners and facilitate the growth of a community of practice. Therefore, the training would involve role-play, peer assessment, and the telling of personal experiences to vest the participants. There was an emphasis on shared ideas for problem solving and feedback to encourage the community of practice. This would help the LoMCId team to achieve high standards of customer service and performance improvement outside of the training environment.
Motivation to Learn The LoMCId team members were highly motivated to do what is “best for the clientele” and took pride in their work. The activation phase of training would include elements to (1) reinforce the LoMCId mission, (2) share reasons for working at LoMCId, and (3) articulate how LoMCId created the new Service Standards. The BSU ID team also provided a job aid.
PreferredLearning Methods Employees and volunteers both preferred interactive training methods rather than lecture. Short scripted scenarios would demonstrate exemplary and undesired behaviors. Participants would critique the behaviors modeled and provide feedback. Then in groups, the participants would practice customer service skills in a variety of role-play exercises. Each participant had the opportunity to play both the clientele and the staff member. The scenarios would reflect on-the-job situations identified as current issues.
Language One learner identified her primary language as other than English, and she noted she could not read English well. To accommodate all learners, the BSU ID team would keep written work to a minimum.

 

Task Analysis

Method and Rationale
To represent exemplary performance, the ID team decided to use the Critical Incident Technique (CIT) described by Militello and Crandall (1999) because:

  • Meeting the LoMCId Service Standards involved complex decision-making in a stressful, fluid environment
  • There were no concrete processes or procedures
  • CIT identifies the critical elements of job, skills, and tasks by collecting real-world data about them (p. 181)

In addition, the BSU ID team decided that a procedural task analysis focusing on observable behaviors alone would not adequately capture the small nuances involved in meeting the LoMCId Service Standards.

Using the CIT, the BSU team conducted interviews with the directors and staff to generate statements of exemplary behavior. The team provided a rationale for each statement by identifying its related Service Standard(s). The team and the client then rated these statements to indicate their criticality to the training. Criticality levels were:

  1. = The training must include discussion about the statement of exemplary behavior
  2. = The training should include discussion about the statement of exemplary behavior, if time allows
  3. = The training will not include the statement of exemplary behavior. It is better addressed using other solutions or later training, such as town hall meetings or evaluations

From this, the BSU ID team identified two main classifications of behavior that the training needed to address:

  • Responding to the needs of the clientele and team members
  • Communicating with the clientele and team members

The CIT also provided the basis for authentic scenarios that the team would later use to create training demonstration, practice exercises, and assessments. These scenarios closely resembled real-life situations the participants would encounter in their daily work at LoMCId.

Training Portion of Performance Solution
Because of the need to model and coach complex interpersonal skills so that the staff and volunteers could practice them to mastery, the BSU ID team chose instructor-led training with a role-play core. This training employed Merrill’s (2002) first principles of instruction by:

  1. Having the participants discuss their reasons for working or volunteering at LoMCId
  2. Providing a “Striving for Excellence” advance organizer to display how the skills map to the overall goal of reaching “excellence in customer service”
  3. Incorporating immediate feedback and coaching from the facilitators on the small nuances of the skills
  4. Ensuring the participants, through the role-plays, actively practice these skills in a manner that mirrored the objectives of the training and modeled on-the-job task performance
  5. Including an assessment consistent with the objectives of the training that would be conducted by the directors on the job within two to three weeks of the training completion

These elements ensured that the participants possessed the knowledge and skills to implement the Service Standards. According to Clark (2012), the use of Merrill’s first principles allows for an evidence-based approach that is highly important in performance improvement. Clark notes that benefits of using evidence-based performance improvement include “quicker results with more front end effort,” as well as “reduced costs”–both of which were important in this setting.

The training consisted of two units for a total of 3 hours. Its design enables the client to deliver the units individually or in combination to accommodate possible time constraints of future training sessions. Table 4 lists the objectives of the two units.

Table 4. Training Objectives.

# Behavior on the job Conditions on the job Criteria on the job
1 Respond to the needs of clientele and team members
  • While at LoMCId
  • While performing on-the-job duties involving various clients
  • In an emotional and potentially stressful environment
 Following the Service Standards:

  • Assisting appropriately
  • Being proactive and anticipating needs
  • Resolving the needs based on the situation
2 Communicate with clientele and team members
  • When there is a message to be delivered
  • In a busy, frequently changing environment
  •  Choosing the appropriate time and place
  • Maintaining confidentiality
  • Using a professional, respectful, and compassionate tone
  • Tailoring the message to the receiver
  • Accommodating for readiness
  • Using appropriate language

 

The BSU ID team designed the scenarios to be “plug-and-play” with notations that identified whether they were best suited for demonstration or one of the practice or assessment activities, and which skills they involved. The team provided more scenarios than the actual training required. In this way, the LoMCID directors and volunteer coordinator could choose which scenarios to use, based on the current situations at LoMCId. The team also provided a template for LoMCId to add new scenarios as the environment and training needs of the organization changes.

The BSU ID team built an authentic performance assessment that LoMCId used during the activities in training, as well as for a Level 2 assessment on the job after training. This allowed participants to become comfortable with the assessment tool prior to its use for on-the-job evaluations. This tool provided benefits beyond measuring the closure of the performance gap. It also provided a mechanism to give clear, precise feedback to LoMCId team members. In addition, it can illuminate new areas for focus in future trainings and help collect new, relevant scenarios for those trainings.

Formative Evaluation
The BSU ID team facilitated a train-the-trainer session for the director of operations and the volunteer coordinator. From this, the BSU ID team made a few revisions to the facilitator’s guide to clarify the flow of activities, improve the usability of the guide, and provide clear role-play instructions.

The client and the BSU ID team then piloted the training with three employees and three volunteers as the participants. Due to time constraints, participants only completed a Level 1 evaluation. The BSU ID team designed the Level 2 assessment to be administered on the job by the LoMC director of operations and the volunteer coordinator within two to three weeks of the pilot.

According to the Level 1 data, the participants reacted favorably to the training. However, analysis from this evaluation indicated a few necessary revisions. These included reducing the number of checklists in the participant packets, changing the headings of the role-plays to make them easier to recognize, and printing the checklists on color-coded paper based on the unit in which the participants would use them. The BSU ID made the first two revisions and gave the last one as a suggestion to the client for the next time they run the training. The BSU ID team also determined that the facilitator guide would be more useful if broken into two manuals: one step-by-step for facilitation and one with explanations and tips for pre-training preparation. However, this modification lay beyond what the schedule would permit.

Recently, the LoMCId held its first formal training sessions, and initial feedback from the client indicated that the participants found the training to be valuable. “Word is getting out to volunteers and board members from people who have taken it what a good resource it is…” (excerpt from client-provided email). The client has also stated that they have been unable to get the Level 2 assessments completed yet.

The BSU ID team has formed an ongoing relationship with LoMCId to provide support, guidance, and feedback for future trainings to ensure the success of the program. It is possible that one member of the BSU ID team will do summative evaluation of the training once it is fully implemented. Should this occur, the authentic Level 2 assessments could be repurposed with little effort to collect Level 3 data indicating skill transfer to the workplace.

Other Components of Performance Solution
Because of the multiple causes of the performance gap, training alone would not close it. Therefore, in addition to the training, the BSU ID team provided LoMCId with recommendations for a total performance solution that addresses the other causes of the gap.

Data:

  • Openly promote the Service Standards by making them visible throughout the organization. This could include hanging posters, putting printed copies around the facility in family areas, and giving printed copies to the families.
  • Encourage peer-to-peer feedback by including stories submitted by volunteer and staff members in the quarterly newsletter, relating to how the Service Standards were used in specific situations.
  • Use the performance assessment tool provided to do annual (or semi-annual) evaluations of the volunteers to help them understand where they are excelling and where improvement can be made.

Resources:

  • Change identified processes that were hindering the delivery of compassionate care.

Incentives:

  • Create a regular column or award that highlights exemplary performance.

Capacity:

  • Develop a placement questionnaire for volunteers to ensure job assignments in positions where they best fit and could excel.

As of the date of publication, LoMCId has implemented most of these recommendations with the exception of the annual evaluations and the placement questionnaire. There is no data available yet on the impact of the total performance solution provided. However, ongoing conversations with LoMCId have showed that they are extremely happy with the results and have discussed submitting the solution to the global office within the next year for a grant award to fully implement it. In addition, either a member of the original BSU ID team or a future team of students may conduct a summative evaluation of the solution. LoMCId have also provided letters of recommendations to the BSU ID team members and have suggested the BSU Instructional & Performance Technology program to other charities for their human performance technology needs.

References
Clark, D. (2012, April). Back to our future: Evidence-based practice for ISPI. Paper presented at ISPI’s Performance Improvement Conference, Toronto, Canada.

Merrill, M. D. (2002). First principles of instruction. Educational Technology Research & Development, 50(3), 43-59.

Militello, L., & Crandall, B. (1999). Chapter eighteen: Critical incident/critical decision method. In D. H. Jonassen, M. Tessmer, & W. H. Hannum (Eds.), Task analysis methods for instructional design (pp. 181-192). Mahwah, NJ: Lawrence Erlbaum Associates.

About the Authors

Stephanie Clark is the managing editor for the curriculum labs for an international computer training company. She will complete her master’s degree in instructional and performance technology in 2013. Stephanie may be reached at stephaniekclark@yahoo.com.
Amanda Collins is the nursing program administrator for a private college in Boise, Idaho. She will complete her master’s degree in instructional and performance technology in 2013. Amanda may be reached at amandacollins2@u.boisestate.edu.
Julie Kwan is a curriculum developer for a global business and IT consulting firm. She will complete her master’s degree in instructional and performance technology in 2013. Julie may be reached at kwan_julie@yahoo.com.
Allison Sesnon is currently completing her master’s in instructional and performance technology at Boise State University. Her interest in performance improvement was spurred from the many performance issues she encountered as a manager in the media and technology sector. Allison may be reached at Allison@performanceval.com.