International
Society of Performance Improvement Newsletter
August 2003
Improving
Performance:
Low-Cost Solutions in a Tight Economy
by
Harold D. Stolovitch, CPT and Erica J. Keeps, CPT
For the last three years, we have repeatedly heard that the economy will turn around “very soon.” Meanwhile, budgets grow tighter and every new training and performance support initiative is scrutinized with a magnifying glass in one hand and an ax in the other. Despite the austerity on the learning and performance support side, the pressure is still on to produce and maintain an increasingly productive workforce and prepare employees for new systems, regulations, and products.
How do we as performance improvement specialists cope with these seemingly impossible challenges credibly? The usual route seems to be via technology—e-learning, reusable learning objects, knowledge management systems, learning management systems/learning content management systems, webinars, and other forms of distance learning. However, we all know these are fraught with uncertainties, frequent lack of user adoption, and early abandonment. There are also the high initial financial investments, infrastructure costs, and time required to develop and implement new technology-based interventions. Worst of all, what results can we anticipate? There is little research to guarantee high return-on-investment (ROI).
Why not turn to low-cost solutions with strong track records of high yield that are much easier to develop and implement? Here are a dozen of our favorites:
Cleaning up performance expectations. Research in human performance identifies a lack of clarity of expectations to be the number one cause of inadequate performance. Despite its high toll on productivity, this cause tends to be neglected when searching for improved efficiencies and results. Yet, reviewing, verifying, and redefining expectations so that they align with desired behaviors and outcomes costs little while producing extremely high ROI.
Developing feedback systems. Inadequate feedback constitutes a close second to unclear expectations in causes of deficient performance. Feedback comes in two useful varieties: corrective feedback that informs performers as to where they are off-target, and confirming feedback that tells them they are doing the job right. Feedback systems are far less costly than virtually any technological intervention and produce powerful impact.
Creating performance support systems. No matter how good the training, without adequate support mechanisms, acquired skills and knowledge tend to deteriorate. Review of procedures, processes, and tasks, guidance in initial stages of application, and emotional support to build and maintain confidence all provide inexpensive support that results in fluent performance.
Designing simple and effective job aids. This is still one of the most cost-effective means for achieving performance results quickly. Decision tables, algorithms, step-by-step procedures, directories, cookbooks, troubleshooting guides, and checklists are some of the many job aid options. When well designed, they result in immediate success. If training is required, it is usually brief to help performers use the job aids.
Ensuring adequate policies, processes, procedures, and resources. When the roadmap is unclear, one is bound to get lost, especially if the vehicle is also faulty. Review of policies, processes, procedures, and adequacy of resources is a first step. Stakeholder consensus, support, and consistent application rules are the next steps. Communication, practice, feedback, and enforcement constitute the final steps. Compared to high-tech interventions, this is a low-cost, high-yield solution to obtain desired performance.
Providing and aligning incentives and consequences. In a recent year-long, rigorous study, we discovered that incentives, especially tangible ones, can improve performance 13-40% (Stolovitch, Clark, & Condly, 2002). Well-designed incentive systems and well-administered consequences that are perceived as fair and equitable can have dramatic results on performance.
Increasing motivation to perform. Motivation is critical in both learning and performance along with ability and prior knowledge. By influencing perceived value to perform and confidence in ones ability to succeed as well as eliminating factors that create negative mood, we can strongly affect workplace motivation. Motivated workers tend to perform at much higher productivity rates.
Making sure mechanisms for training transfer are in place. Studies have shown that several months after a training intervention, usually less than 30% of what was learned remains. Simple interventions such as having supervisors prepare workers for training, ensuring that training includes considerable practice closely aligned to back-on-the-job work, and support and follow-up post-training immensely increases on-job transfer.
Reviewing and tightening performer selection systems. Poor selection of performers results in poor performance. Select individuals with characteristics, prerequisite skills, and knowledge that you are not prepared to invest in. Train for specific, job-required competencies. By tightening selection criteria and procedures, you can better fit the right person to the right job. This decreases training investments, supervision time, and turnover—all costly items in a tight economy.
Eliminating tasks that interfere with job performance. Lower priority or administrative tasks often interfere with getting important jobs done. For example, a salesperson with a specified number of calls to make or amount of revenues to generate may be held back due to mandated report writing and meeting attendance. Eliminating unnecessary, counterproductive tasks can result in considerable performance improvement.
Adopting structured OJT and PAL. More money and time are spent in informal on-the-job training (OJT) than in formal classes, including e-learning—perhaps four to five times as much. Imagine the efficiencies if OJT were structured. Peer Assisted Learning (PAL), in which more advanced peer-level buddies are trained to assist novice workers, can also achieve remarkable results at a relatively low cost.
Strengthening existing training. When all else fails, review and clean up existing training? You can retrofit ineffective training by applying sound instructional design to it. Break existing training into logically connected units. Then, build in meaningful rationales, objectives, learning activities, evaluation, and feedback. By strengthening your current training instructionally, you can increase its effectiveness at a fraction of the cost of building new, more sophisticated learning systems.
Tough times call for creative cost-cutting measures. Lets get back to the fundamentals of performance improvement. There certainly is a place for technology, but lets not start with expensive solutions until we have clearly defined both the goals and the gaps. Front-end analysis allows us to do this systematically. Then, with a sharp eye on engineering the best performance improvement solutions at the least cost, we can make a significant contribution to achieving remarkable organizational results at low cost in this tight economic environment.
Reference:
Stolovitch, H., Clark, R., & Condly, S. (2002). Incentives,
motivation & workplace Performance: research and best practice. Silver Spring, MD: ISPI and SITE Foundation.
Harold D. Stolovitch and Erica J. Keeps share a common passion—developing people. Together they have devoted a combined total of more than 70 years to make workplace learning and performance both enjoyable and effective. Their research and consulting activities have involved them in numerous projects with major corporations worldwide. Harold and Erica are the principals of HSA Learning & Performance Solutions LLC, an international consulting firm that specializes in the application of instructional technology and human performance technology to business, industry, government, and the military. They are co-editors of the Handbook of Human Performance Technology and co-authors of the best-selling, award-winning, Telling Aint Training. Harold and Erica may be reached at info@hsa-lps.com.
Would you like to advertise in this space? Contact marketing@ispi.org |
|||
TrendSpotters:
Future Watch, Featuring Paul Harmon
by
Carol Haig, CPT and Roger Addison, CPT
This month we share predictions that affect all performance improvement practitioners, made by a long-term ISPIer whose experience and interests led him to focus on software technologies and business process change. Paul Harmon is a founder and the executive editor of Business Process Trends, an information portal and web ’zine aimed at business process change managers and information technology specialists and author of the book Business Process Change. He may be reached at pharmon@sbcglobal.net.
In Pauls view, business process refers to everything from strategic organizational goals and policies to the automation of specific activities, and all of the tasks in between. The people who work with supply chains, value propositions, information technology, Six Sigma, etc., all need to connect through their business processes to identify trends, directions, and best practices for their organizations.
Top
Three Predictions
First, Paul predicts that the transition to the process-centric organizational
structures we are starting to see will continue and accelerate in the
next two to three years. The resulting new organizational designs will be matrix-based
to encourage collaboration and information sharing.
Second, Paul predicts that on-going process change programs will emphasize human performance improvement. Information technology is expensive to develop and carries a high risk factor. In the past, the focus was on the latest software technology and making it work. Now, many senior leaders realize that it is easier to change the processes and the people responsible for them than it is to fix an automated system. “People technologies” often give us more for our investment dollar than do new automated technologies, especially in recessionary times like these. As organizations see results from human performance improvement initiatives, they will further emphasize this element in their process change activities.
Pauls third prediction is combined with his hope that ISPI will reach out to new audiences in different functional areas to show how human performance technology (HPT) applies to them. He cites a number of highly competent, recognized practitioners in ISPI whose work is focused on process improvement, such as Geary Rummler and Don Tosti in the United States, Klaus Wittkuhn in Germany, and Artur Nunes in Portugal. It is time for all ISPIers to step away from talking only to each other and bring their HPT tools and techniques to other parts of the organizations they serve.
Why
These Predictions?
Historically, systems werent built to talk to each other because the
technology wasnt there and the cost was too high. Then the Internet was
born, making it technically possible and economically feasible to tie business
processes together.
Today, organizations can think broadly about a specific application and identify all the parts of the business process that the application touches, making the transition to a process-centric organizational structure a logical move. Indeed, systems are being refitted across all industries. And this work is focused on processes—how best to connect the pieces to serve the customer or tie in the supplier.
Process change programs will emphasize human performance improvement. The Gartner Group, the leading information technology research company, reports that even today only 20% of the activities performed in an organization can be automated. This means 80% of decisions and steps in an organizations processes are still done by people. Senior managers have learned that slick, smooth processes must include people along with the automation, bringing people-processes to the forefront of all design work.
Like many of us, Paul observes that ISPIers talk about processes and teach each other, but dont reach out to other disciplines. We need to change this if we are to truly make a difference in the workplace and remain a viable professional Society. As Klaus Wittkuhn says in his response to the February 2003 special Performance Improvement journal issue on the HPT value proposition, “ISPIs value proposition should target decision makers and line managers.”
At this time, we have an extraordinary opportunity to move our technology out to the business process level in our organizations. We, in performance improvement, are uniquely positioned to do this right now. If we work in the training department, we usually have clients throughout the company, and we know the projects that different functions are working on. We can connect clients working on similar projects and help them use performance improvement tools. Such activities reduce the duplication of effort that silos foster and pave the way for cost reductions. And thats how HPTers can add real value.
So
What Do We Do Now?
Organizations are shifting to a business process structure for all the reasons
described above. As they re-align around business processes, opportunities
to use HPT tools and models can only grow.
And so we put these critical questions to you, our readers: What should performance improvement practitioners do in response to such changes? How do we make ourselves proactive and teach our clients to use our tools? If our organizations are currently describing the same problems and opportunities from different silos, how do we get them to open a dialog from a business process perspective?
What do you suggest we do, individually and collectively? Email either Carol or Roger at the email addresses provided below, and we will publish your ideas in a future TrendSpotters article.
If you have any predictions about the future of HPT that you feel would be of interest to the PerformanceXpress readership, please contact Carol Haig, CPT, at carolhaig@earthlink.net or Roger Addison, CPT, at roger@ispi.org.
Is
Your Training Organization Performance-Based?
by
Ann W. Parkman and Karen VanKampen
One question many training professionals are asking themselves is “How can we directly impact our organizations profitability?” This question underscores the transition companies are making from traditional training to performance-based training.
What does it mean to be performance based? It means your departments primary focus is on improving job performance as a means of directly impacting the quality of your organizations products and services and the efficiency with which they are produced. To achieve this, the following critical elements must exist:
Here are some recommendations to consider when assessing ways to transform your department into a true performance-based training organization:
Focus
on Results
Your departments focus should be on improving job performance and
sustaining desired performance levels. To maintain a results-oriented focus,
ask yourself
if your organization:
Training
Methodology
Adopting a performance-based methodology such as CRI will ensure that training
provides the job-critical skills learners need to meet managements expectations.
Here are some questions to help you determine how your training organizations
methodology compares to CRI:
Performance
Measurement
Performance-based training organizations are in the best position to influence
the development of a plan for standardizing and aligning performance measures.
To help you assess how your training organization measures up, ask yourself:
If you answer “no” or “dont know” to any of these questions, you may need to close some gaps between your current training focus and your desired performance improvement focus.
Ann W. Parkman is executive vice president, managing partner, and co-founder of CEP (The Center for Effective Performance), as well as a past president of ISPI. She will be facilitating a session at ISPIs Fall Conference entitled, “How to Recognize Performance-Based Training When You See It.” Ann may be reached at aparkman@cepworldwide.com.
Karen VanKampen is director of performance consulting for CEP. She has spoken at a number of ISPI conferences and local ISPI chapter meetings. Karen will be facilitating a session at ISPIs Fall Conference entitled, “Becoming a Performance-Based Training Organization: Beyond the Basics.” She may be reached at kvankampen@cepworldwide.com.
Would you like to advertise in this space? Contact marketing@ispi.org |
|||
OQ:
Your Mother
by
Sivasailam “Thiagi” Thiagarajan
In the July issue of PerformanceXpress, I presented an online cafe (or palaver hut) as a remedy for the mindless multiple-choice questions that crowd web-based training materials.
This strategy, called Open Questions (OQ), presents a question that permits more than one acceptable answer. OQ requires and rewards reflective and creative thinking. It also facilitates mutual learning by providing related webpages for you to explore:
Results
We kicked off the inaugural OQ with a reflective question (“What are
the advantages and disadvantages of the OQ approach?”) and a creative
question (“If GAME were an acronym, what does it stand for?”) Within
24 hours, we were inundated with enthusiastic responses from several PX readers.
Answers are still trickling in. We have cleaned up the clutter and summarized
the key points.
You can check out the responses about advantages and disadvantages of OQ by clicking here. To enjoy creative expansions of GAME, visit here. Feel free to contribute your answer for these earlier OQs. As several readers have pointed out, OQ permits continuous improvement of your mastery of the topic. So write a new and improved answer even if you contributed an answer earlier.
A
New OQ
Heres an open-ended question that has been around ISPI circles for several
decades: “How would you explain HPT to your mom?” Click here to
check out the OQ pages and contribute your answer.
As an extra incentive, I will have a panel of judges select the “best” answer on August 15, 2003. We will publicize this answer in a future issue of PerformanceXpress.
From
the Board:
Chapters...ISPI’s Lifeline to the Frontlines
by
Clare Elizabeth Carey, CPT, ISPI Director
As an ISPI Director, one of my key responsibilities is to provide updates on the progress of my assigned committee and/or task force. As Im writing this article by my pool, on another perfect 85 degrees, crystal blue sky, gentle winds day in Hawaii, I cant help but smile. But it is not because of the weather or my lush tropical setting. Its because I have the good fortune to be designated as the Board Liaison to the Chapter Partnership Committee (CPC). I say, “good fortune” with all sincerity. The CPC is an engaging and fun group who brings out what is best in ISPI: results, research, replication, and recognition packaged in a professional and collegial network.
Paul Cook leads the CPC leadership team whose members include: William Dudeck (chair elect), Jordan Brun (communications chair), and Dan Topf (former chair). The “Core 4” are supported by the energetic contributions of Roger Chevalier (ISPI), Kim Kidwell, Mia Logan, Robyn Skarbek, Sharon Dwyer, with additional wisdom provided by Diana Vansickle, Jim Wright, Stephany Prodromides, Andreas Kuehn, and James Andrews. This diverse collection of talented HPTers, invests in the continuing growth and development of our chapters while strengthening relationships with International. This is a formidable challenge as our Society increases its global scope amidst austere financial conditions.
This year the CPC will focus its energies on the following goals:
Conduct the Chapter Leaders Workshop: The plan is to make the 2004 Chapter Leaders Workshop bigger, better, and more valuable to chapter representatives. This full-day event designed specifically for the needs and interests of our chapter leaders is offered at no cost to participants. All ISPI members are welcomed and encouraged to take advantage of this conference opportunity, as well.
Promote and support greater use of the CPC website: Given ISPIs global community, the Internet highway is the road of choice for our members. The CPCs dedicated webmaster (Kim Kidwell) works to maintain currency and accuracy of information posted on the CPC website, as well as expanding the Intranet linking of chapters.
Promote and share the best practices of chapters: The CPC promotes the application of the Award of Excellence criteria as a strategic mechanism to guide local chapters. Chapter of Excellence recipients are useful sources of best practices and serve as mentors and benchmarks for other chapters.
Improve communication between CPC and local chapter leadership: Communication flows in multiple directions and languages. In addition to publishing its own “Almost Monthly Newsletter,” the CPC has initiated the Chapter Leader Forum. This e-mail discussion group supports the mission of ISPI and the successful operation of chapters. The CPC team encourages the free exchange of ideas and perspectives and the application of HPT to solve chapter problems.
Improve the CPC as a resource for local chapters: Responding to the feedback from chapter leaders about their greatest challenges, the CPC provides links on its webpage to many resources including the Chapter Management Handbook, newsletter articles, program schedules, available presenters, and successful marketing strategies. Emerging chapters do not need to struggle to create their infrastructure or operating procedures. There is much that has already been done by other chapters—with successful results. The CPC is working to eliminate redundancy and the normal day-to-day frustrations with local chapter operations. They recognize chapters are volunteer organizations with limited time and resources. By harnessing the available CPC resources and references, local chapters can save enormous amounts of time and capitalize on the success of their sibling chapters.
As a conduit for performance improvement, the CPC capitalizes on its collective expertise to share information, tools, and resources among ISPIs 75 chapters around the world. The CPC has a global view. They want to “internationalize” committee representation, products, and activities. Overcoming geographical boundaries and time zone challenges, the CPC leadership team conducts regular conference calls to plan, organize, and delegate its tasks. They are focused in their efforts and optimistic in their charge. They believe in ISPIs mission, they value our Society, and they want to serve as lifelines to ISPIs frontlines.
I cant help but smile. Its so refreshing to witness passion in action.
I-Spy:
Websites of Interest
by
Todd Packer
Ah, summertime, and even performance technologists deserve a vacation. Through our “I-Spy” column, we hope to offer some useful leisure-time reading for our readership through relevant, interesting, and useful websites for performance technologists. Each month, we take readers to off-the-beaten-path sites that help them find similar thinkers, resources, work, new ideas, and sometimes just plain old fun.
Quick recap: Every month, three sites, one theme. While far from comprehensive, hopefully these sites will spark readers to look further and expand views about human performance technology (HPT). Please keep in mind that any listing is for informational purposes only and does not indicate an endorsement either by the International Society for Performance Improvement or me.
These are the general categories I use for the sites featured:
The theme for this months column is Holidays. In many places, August is a time for vacation, travel, and holidays. Allowing yourself some time to reflect and renew can improve your personal and business performance. So, take a deep breath, read this PerformanceXpress issue, then turn off your computer and plan for some time away from work. Dont forget to pack your toilet paper.
E-Klatch
For some interesting links to information for business training professionals
on the go, take a trip to the website of the Association
for Development, Advancement & Productivity through Technology Training (ADAPT) “a
non-profit organization providing information technology training professionals
with a forum to exchange ideas and expand industry knowledge.” Under
the resources link, you can find training
and learning organizations across the U.S. (yes, ISPI is listed!),
connections to some periodicals for
travel reading (such as Business Travel News), job
postings, and links to
help you with everything from airfares to weather.
HPT@work
Are you ready to plan some time away from your work? A visit to several sites
designed by Purdue University Professor Alastair
M. Morrison, PhD can help. Dr. Morrisons resources can be quite
valuable, whether youre looking for work (Hospitality
Travel Hotel Tourism Jobs and Employment on the Web—contains a very
extensive set of links for countries around the world), or youre
seeking ideas for planning a sabbatical/time
off from work (Sensational Sabbatical Suggestions). And, if youre
still having difficulty justifying your travels, take along an academic
journal (such as Journal of Quality Assurance in Tourism & Hospitality or Visions
in Leisure and Business) for some reading at the beach.
I-Candy
Well, if you would like to travel, but cant seem to tear yourself away
from the computer, pay a visit to the Virtual
Library of Museums around the world. Youre likely to find a unique
museum to meet your interests, including The
Baghdad Museum Project (a virtual museum of the Iraq National Museum),
the Te Papa Tongarewa
Museum of New Zealand, and Frances Paleolithic
Cave of Chauvet-Pont-DArc. And, now you can really take a break as
you review 4,500 years of sanitation at Indias Sulabh
International Museum of Toilets.
If in your journey across cyberspace, you find other websites with beautiful travel pictures or that may be of interest to PX readers, email your finds to I-Spy.
Until next time, best wishes for safe travels across the globe or the keyboard. See you in our September PerformanceXpress!
When he is not Internet trawling for ISPI, Todd Packer can be found improving business, non-profit, and individual performance through research, training, and innovation coaching as Principal Consultant of Todd Packer and Associates based in Cleveland, Ohio. He may be reached at tp@toddpacker.com.
Measurement
Counts!
Join the GOT RESULTS? Campaign!
by
Carl Binder
Im pleased to announce the “new” GOT RESULTS? Campaign that Timm Esque, Julie Capsambelis, I, and others in ISPI are about to launch. Were excited about the prospect of it becoming an even more valuable contribution to ISPI and the field of human performance technology (HPT).
A
Little History
If youve attended ISPIs Annual Conference in the last two years,
you have probably seen the GOT RESULTS? exhibits. They have showcased several
dozen examples of performance systems or HPT interventions that produced results
in the form of directly measured improvements in behavior, job outputs (accomplishments),
or business outcomes. The exhibits have been part of an effort that goes back
to the mid-1990s when Ogden Lindsley (1999) observed how few actual performance
data make their way to ISPI publications. That observation inspired Timm Esque
and Pat Patterson to publish their book, Getting
Results: Case Studies in Performance Improvement, and prompted a small
group of us to begin formulating strategies and tactics to increase the number
of measured results captured and shared among HPT professionals and ISPI members.
From
an Event to a Campaign
The GOT RESULTS? exhibits have made a public showing of some great case studies,
and we are very proud of the effort so far. Most of those cases have been posted
as PDF files on ISPIs GOT
RESULTS? webpage. Our objectives have been to provide practitioners with
examples of practical measurement, and to demonstrate with data that HPT methods
produce valuable, measurable outcomes.
After the 2003 conference exhibit, we had discussions with a number of ISPIs officers and thought leaders and began to shift our vision for GOT RESULTS? from a conference-based event to an ongoing campaign in which we will accumulate an expanding variety of results case studies and build an informal network of HPT practitioners skilled in measurement and interested in helping others to measure and share the results of their own work. We emphasize the word practitioner, because this is not about academic research. It is about practical measurement and communication of valuable performance outcomes.
With this new vision, we will still have an exhibit at the conference. But our new emphasis will be on building an archive and network of practitioners to serve the promotion, education, and marketing of HPT and ISPI.
A
Performance System
My colleague, Timm Esque, helps his clients establish performance systems in
which the most essential element is ongoing measurement of results and databased
decision-making aimed at improving outcomes (2002). We envision the GOT RESULTS?
campaign as a performance system in which we will measure how many promises
to submit cases, how many cases actually submitted, and how many people join
our informal network of measurement-based practitioners per month. Well
use those data to adjust our strategies and tactics over time, with the goal
of maximizing involvement of our colleagues and submission of GOT RESULTS?
cases.
Get
Involved
There are many ways that you can get involved. You can check out the revised webpage.
You can download the new submission
form and be on the lookout for opportunities in your own work or in the
work of your clients and colleagues, for measuring performance in ways that
will satisfy the GOT RESULTS? criteria. Also, keep an eye out for publications
or conference presentations with results that would fit our criteria, and then
help make the connection so we can include those cases in the ISPI archive.
And finally, if youd like to help in the overall campaign, please contact
me by email.
Over the coming months, well be enhancing the GOT RESULTS? webpage, linking it to more places on the ISPI website, and doing everything we can to solicit cases. Please join us in helping to build the GOT RESULTS? archive and network of practitioners.
References
Esque, T.J. (2002). Making
an impact: Building a top-performing organization from the bottom up.
Atlanta, GA: CEP Press.
Esque, T., & Patterson, P.A. (Eds.). (1998). Getting results: Case studies in performance improvement. Amherst, MA: HRD Press, Inc.
Lindsley, O.R. (1999). From training evaluation to performance tracking. In H.D. Stolovitch and E.J. Keeps (eds.). Handbook of Human Performance Technology, second edition. San Francisco: Jossey-Bass/Pfeiffer, 210-236.
Dr. Carl Binder is a Senior Partner at Binder Riha Associates, a consulting firm that helps clients improve processes, performance, and behavior to deliver measurable results. For additional articles, visit http://www.binder-riha.com/publications.htm.
How
Effective is Training? A
New Summary of the Past 40 Years of Training Field Research and Evaluation
by
Dick Clark, CPT
Many training studies have been conducted in the past 40 years with conflicting results. A new summary, published in the Journal of Applied Psychology, gives the most accurate view to date of the evidence for the average impact of training found in all adequately designed and published field studies from 1960 to 2000 (Arthur, Bennett, Edens, & Bell, 2003).
The authors of the summary, researchers from Texas A&M University and the Air Force Research Laboratory, reviewed all 636 studies published in the past four decades and selected only those that were adequately designed. They limited their review to studies that had actually been conducted in organizations (they eliminated all “laboratory” studies from their review). They subjected the results of the 162 studies (26% of the total) reporting 397 comparisons that survived their design review to a statistical process called “meta analysis.” This process is thought by many to reveal the most accurate “average” impact of a performance improvement strategy such as training. The authors used the Kirkpatrick four-level system for categorizing the results of the studies they summarized and averaged.
The quick bottom line of this huge review is that, on the average, training has resulted in an approximately 20% performance increase at all of the Kirkpatrick levels.
You will find the article that reports the study in detail in a good university library, but until you have the opportunity to read it yourself, here are some of the findings they report:
The average learning (level 2) impact of training is a 20% increase in knowledge and skills. So if the average trainee received a score of 50% on a performance test before training, they would have received about 70% on the same test after the training.
Only 7% of studies report level four impact results—but in those studies level four results were a very impressive 18%. This can be interpreted as indicating that the division or organizational performance that was supposed to be improved by the knowledge and skills that were transferred to the job increased 18% as a result of training.
Level three (transfer of training to work settings) impact was also about 19-20%. This implies that in these well-designed studies that presumably reflect the best training models, most of what was learned was transferred back to the job and applied.
The most effective training content was interpersonal skills (such as listening, teamwork, communication) and resulted in 27% learning—but when a cognitive approach to interpersonal skills was used, the result was a whopping 64% increase in learning and a 25% transfer level.
Only 6% of the studies reported that a needs assessment was conducted before the training was designed and implemented. Since these studies probably reflect the best training programs (people are not inclined to conduct experiments on programs they do not trust), and since people are inclined to report a needs analysis when it has been conducted, it is doubtful that needs analysis is well established in the training community.
When the researchers asked what training medium was associated with the highest levels of learning, transfer, and impact, the winner was clearly “live instruction” by a trainer (they called it “lecture”)—a 44% increase in learning.
It is interesting that 6% of the published studies reported a negative result of training on learning, transfer, and/or impact. This means that some training programs significantly decreased learning, transfer, and impact. Even more interesting is the authors claim that this percentage should be about 10% and that the missing negative impact studies suggest that when experiments find negative impact, they are not often published. This gives the wrong impression that training almost always has a positive result when about 1 study in 10 is negative.
Finally, there seemed to be no “fall off” or “decay” of training results over time. In the studies where delayed measurements of the four levels were obtained, the results of training seemed to hold up over time.
One might ask whether the studies that were reviewed are representative of the training that is currently being implemented in todays organizations. I constructed a quick frequency count of the percent of studies that investigated each of the four Kirkpatrick levels and compared the percentages with those reported last year in the ASTD training survey (2002). You can see the results in the table below.
| Arthur et al. | ASTD | |
| Reactions | 4% | 78% |
| Learning | 59% | 32% |
| Transfer | 31% | 19% |
| Impact | 7% | 7% |
Table
1. Percent of Kirkpatrick Levels studied
in Arthur et al. and reported by ASTD.
You can see that many more training organizations report collecting reaction data than was collected by the authors of the studies that were reviewed. Yet it is possible that the researchers conducting the reviewed studies reasoned that reaction data were less important than learning, transfer and impact data. Notice that the frequency of the “bottom line” data is exactly the same in the two samples and that the learning and transfer data is closer to the averages reported in the ASTD survey. These comparisons lead me to conclude that the studies reported in the review might be representative enough of the training that is currently taking place.
References
Arthur Jr., W., Bennett Jr., W, Edens, P., & Bell, S.T. (2003). Effectiveness
of training in organizations: A meta-analysis of design and evaluation features. Journal
of Applied Psychology, 88(2), 234-245.
Van Buren, M.E., & Erskine, W. (2002). The 2002 ASTD state of the industry report. Alexandria, VA: American Society of Training and Development.
Dick Clark is the principal of Atlantic Training Inc., a company that offers consulting on training design and performance improvement strategies. He is also a professor of Educational Psychology and Technology in the Rossier School of Education at the University of Southern California where he serves as the mentor of USCs doctoral program emphasis in Human Performance at Work. Dick received the Thomas F. Gilbert Award for Outstanding Professional Achievement from ISPI last year. He and Fred Estes recently authored Turning Research into Results: A Guide to Selecting the Right Performance Solutions (CEP, 2002). Dick may be reached at clark@usc.edu.
Message
from the President:
Board Reaches Halfway Point
by
Guy W. Wallace, CPT, ISPI President
The third Board of Directors meeting, of the six in total, was held in San Francisco, July 18-20. It was the halfway point in the year for this Board, and some of us are already feeling a bit of sadness regarding the “end being near.” ISPI Board experience is like that—I know, as this is my second time hitting the mid-point. Ahhh.
But this also reminds me to encourage you to serve our professional home on committees and task forces, at the Board level, as a Director, and then as the President-elect and President. Not for your own ego gratification, which these experiences can indeed offer, but for your own development, both professional and personal.
I have personally learned so much from these experiences over the past 24 years of my active membership that I cannot imagine where my career would be without the insights, knowledge, skills, and friendships that ISPI has afforded me. Talk about a Value Proposition! Click here to check out the nominations section on ISPIs website for more information including position descriptions and qualifications criteria. The deadline for nominations for the ISPI Board of Directors is August 29, 2003.
But I have digressed.
Executive Director Rick Battaglia, Treasurer Barbara Gough, and I were joined by Past President Jim Hill for a pleasant round of golf on the Thursday prior to the Board meeting. I am sure he has some of those pangs, missing the dialogue, the action, and the learning take-aways from Board meetings. It was great catching up with Jim, who continues his involvement with ISPI and his advocacy of HPT in his post-presidency period.
Thursday evening included a dinner meeting with long-time ISPI/HPT advocate Paul Harmon—check out his interview with Carol and Roger featured above in this months TrendSpotters. We discussed his involvement and thoughts regarding both Six Sigma and SCOR and HPT. Paul worked for Tom Gilbert and Geary Rummler in the 1960s and with ISPIs President-elect Don Tosti in the early 1970s. He has deep roots in HPT.
The Board meeting began on Friday morning with the standard “check-in” updates from the Board regarding their personal and professional lives (we each have both!). As it had only been one month since our last meeting, we moved quickly into updates regarding Society happenings, and then got into the meat of the agenda.
The major topic was the continuation of the development of the ISPIs Strategic & Operations Plan, which we are deliberately “not rushing through” and taking (what seems like) the whole year to complete. My hope is that we will provide the Society with an overview of the details sometime in September or October.
Other items addressed included the review of the initial action plans and budgets from those committees and task forces that met our window for review and reaction prior to finalizing both for the September Board of Directors meeting, where we will lock down the budgets for the next fiscal year. Until the October PX Cheers!
2004
ISPI Annual Conference:
Partnering for Performance
Call for Proposals
Amazing! Where did the summer go? School will be starting soon or already has in some locations. Its time to get organized for fall. And chief among the “Things to Do” is to make sure you have submitted a proposal for ISPIs 2004 Annual International Performance Improvement Conference & Exposition, April 18-23 in Tampa, Florida. The deadline is September 15, 2003. Click here to download a PDF of the submission guidelines.
Not sure how to compile a winning proposal? Click here for examples of a sample proposal application template, a successful session proposal, and a sample handout and performance tool.
Volunteer
Opportunities: Calling All Students!
Are you interested in attending ISPIs 2004 Annual Conference & Exposition
but unable to afford the conference registration fee? If you are willing
to attend pre-assigned sessions or workshops, are open to monitoring sessions
you may not have selected on your own, and are able to distribute and collect
evaluation forms and assist ISPI presenters, send your name, complete mailing
address, phone, fax, and email address to: ellen@ispi.org
ISPI will significantly reduce the conference registration fee for all conference volunteers. Volunteers will be responsible for their own travel, hotel, and other costs associated with attending the conference. Volunteers are assigned on a first-come, first-served basis. Full-time students are strongly encouraged to apply and will be given priority status. ISPI will contact you regarding your assignment in October.
2003/2004
Awards of Excellence Program
Who receives an ISPI Award of Excellence?
Cingular, Capital One, CEP Press, University of Toyota, Walgreens and members like you!
Submit your ideas, innovations, programs, or training tools by the October 24, 2003 deadline, and you could be on your way to earning the recognition you deserve. For more information or to download the submission packet, click here or contact Ellen Kaplan, ISPI Senior Director of Meetings at 619.224.4900.
Lean
Instructional System Development in
Engineering the First Successful F-16 Mission Training Center
by
Conrad G. Bills, PhD
I have found that Instructional System Development (ISD) has a broader perspective than what is traditionally taught in instructional technology curriculums. I summarized this expanded vision in my response “Expanding our Vision of ISD in Performance Improvement: Organizational Perspective of Instructional Systems” to the February 2003 special issue of Performance Improvement journal.
By taking an organizational approach to ISD, it becomes a proactive model. By applying a proactive ISD model from an organizational perspective provides a method for making new capabilities have a “good fit” into the instructional system rather than forcing capabilities to be “shoehorned” into an existing model. When users attempt to use the shoehorn approach, more often than not the activity creates complications that impact cost, schedule, or performance. This expanded approach is also responsive to Roger Kaufmans plea to flow our methods so they are responsive to the broader “organizational performance and occasionally to external clients and society” (2003).
Lockheed Martin used the ISD process (Figure 1) to properly integrate with the systems engineering process during the F-16 Mission Training Center (MTC) development. The design was to ensure the training system provided the in-depth instruction and high transfer of skills, knowledge, and behavior needed to improve mission performance. This approach overcame the historical void between requirements definition and operational testing. Early in the design phase of the program, the Training Capability Requirements Assessment (TCRA) process confirmed desired operational training capability. Throughout TCRA, the ISD methodology was applied to ensure achievement of the customer-defined training task list (TTL) and training missions.

Figure 1. ISD Process
The F-16 Mission Training Center (Figure 2) is a four-cockpit simulation system on a local area network. Each cockpit is full fidelity and has a full field-of-view visual environment. In simple terms, the F-16 MTC is a high-end pilot training system that employs computer-generated imagery. The pilot flies in a synthetic combat environment, which allows insertion of simulated friendly and opposing forces from manned threat stations. The geospecific, phototexture database supports accurate air-to-air and air-to-ground operations. As Distributed Mission Operations (DMO) expands to a wide-area network, other remotely located players will join in to provide multiship, multiplatform warfighter interaction without the real-world environmental restrictions or range limitations.

Figure 2. F-16 Mission Training Center
The customer, Air Combat Command (ACC), wanted a training capability to allow them to “train the way they fight.” ACC defined this new dimension of training capability through desired training missions and the corresponding training tasks to be performed. Program schedule and cost constraints limited resources for a formal ISD process. Therefore, Lockheed Martin adopted a “lean” ISD approach. A lean ISD approach meant a reduction in effort while not compromising known principles. Fundamental principles remained constant, and incorporated AF Manual 36-2234, Instructional System Development, and concepts from AF Handbook 36-2235, Vol. 7, Design Guide for Device-Based Aircrew Training.
Rather than further delineating the TTL down to the functional object or cue, our lean approach was to define the mission-training event by a goal, training condition, and performance standard. The team developed representative test mission profiles typical of the Air Force unit tasking for F-16 Block 50 aircraft. Progress in achieving the performance of the TTL was done by calculating the percent of TTL items accomplished against the items assigned, both by mission accomplished and overall. After installation, pilot satisfaction was tracked based on both system training capability and reliability. Results have consistently been high.
Customer authorization for training services was achieved the first time, “out- of-the-chute” through the application of a lean ISD approach integrated within the systems engineering process. The customer, contractor, and end user came together on integrated product teams (IPTs) that functioned together throughout the acquisition. Close working relationships in these multidisciplinary IPTs kept the focus on meeting the needs of the end user.
Highly qualified F-16 pilot subject matter experts (SMEs) were involved throughout the program. Due to manning constraints and operations tempo, Air Combat Command provided periodic participation of pilots from the operational community. Key SME involvement ensured that validation and performance evaluation was in line with operational training missions.
Mission-oriented testing began where subsystem acceptance test procedures (ATPs) ended. Qualified F-16 pilots, who were also familiar with the intended “training conditions” for each TTL item, conducted mission-oriented testing. This lean approach allowed the MTC team to thoroughly “wring out” the system without the use of volumes of material that make up traditional test plans. Even though it was a lean approach, the process easily accommodated development and integration of complex models and interfaces such as the full HARM Targeting System (HTS) simulation. It also helped with delivery of early single-ship training capability.
This lean ISD approach integrated with systems engineering has shown to be sound and is being used for F-16 MTC training capability improvements resulting from aircraft upgrades and simulation technology insertion. Application of lean principles to ISD is not only important to achieving success, but it is a necessity for meeting performance-based requirements in a tightly scheduled procurement.
Reference:
Kaufman, R. (2003). Value,
value, where is the value? Performance Improvement, 42(2) 36-38.
Conrad G. Bills is training manager for F-16 Training Programs. He serves on multiple training system programs in ISD for Lockheed Martin Naval Electronics & Surveillance Systems, Akron, Ohio, covering all phases of the ISD process. He is responsible for the F-16 MTC pilot training system concept, and his understanding in the Training Systems Integrated Product Team was incorporated during the concept phase of the Joint Strike Fighter (JSF) Program. He joined Lockheed Martin in 1993 upon his retirement from the U.S. Air Force. Conrad holds a PhD from Kent State University. Conrad may be reached at conrad.bills@lmco.com.
Call for Nominations to the 2004-2006 ISPI Board of Directors
It is time once again for you, the ISPI membership, to determine the future direction of ISPI by nominating those members that you feel have the qualifications, experiences and vision to lead our Society. Up for nominations this year are the President-elect and three Board members. They will join the President, two continuing Board members and the non-voting Executive Director who make up the eight-member Board.
The duties of the Board are to manage the affairs of the Society and determine the strategic direction and policy of the Society.
Brief Job Descriptions
President-elect
The President-elect assumes the Presidency of ISPI for a one-year term at the
conclusion of his/her one-year term as President-elect. The President-elects
efforts are directed to assuming the Presidency and assignments are designed
on preparation for that transition. The President-elect serves to provide
continuity of programs, goals, objectives, and strategic direction in keeping
with policy established by the Board of Directors.
Director
Each Director on the Board serves a two-year term and is a leader in motivating
support for established policy. He or she serves to develop new policy
and serves to obtain support for ISPIs programs. A director should
provide an objective point of view in open discussion on issues affecting
the membership and profession. He or she should thoroughly analyze each
problem considered, vote responsibly and then support those actions adopted
by majority vote. Individually, each member of the Board is considered
a spokesperson for ISPI and represents the integrity, dedication, and loyalty
to established policy.
The deadline for nominations is August 29, 2003. If you would like to nominate someone, please send the your name and contact information along with the nominees name and contact information to april@ispi.org. Be sure to indicate the Board position you are submitting for in the Subject line. If you are interested in additional information on the nominations process or the complete job descriptions and qualifications required, click here.
CPT Deadline Approaches: Apply Today!
The Certified Performance Technologist (CPT)
designation is awarded by ISPI to experienced practitioners in the field
of performance improvement and related fields such as instructional design
and organizational development whose work meets the 10 Standards of Performance
Technology and other application requirements.
Until October 31, 2003, performance improvement professionals with more than six years of experience can apply under Grandparenting application provision for the CPT designation. After that date, anyone seeking the designation will have to apply under the more rigorous Regular application provision. For more information on the CPT designation and to download the application forms, visit www.certifiedpt.org.
ISPI to Publish HPT Success Stories
Write and submit a one-page story by August 15, 2003, and you can be published in an upcoming ISPI collection of human performance technology (HPT) success stories. This short, performance improvement book will give readers examples of successful applications of HPT, as well as the inspiration needed to apply HPT in the organizations they serve. The end result will be a book that provides a catalog of success stories from the field of HPT. Click here to learn more!
Call
for Submissions:
The ASTD Dissertation Award
The ASTD Dissertation Award is given each year to foster and disseminate research in the practice of workplace learning and performance. This years award will be presented to the person who has submitted the best doctoral dissertation for which a degree was granted between September 21, 2002-September 20, 2003. The topic must focus on some issue of relevance to the practice of workplace learning and performance. Illustrative areas of concentration include: training and development, performance improvement/analysis, career development, organization development/learning, work design, and human resource planning.
All research methodologies will be considered on an equal basis including, for example, field, laboratory, quantitative, and qualitative investigations. The candidate must be recommended and sponsored by his or her committee chair. All materials submitted must be in English and in Word format by email. Submission requirements correspond to the full manuscript requirements of the Academy of Human Resource Developments (AHRD) Dissertation of the Year procedures that require applicants to follow the full manuscript conference proposal submission guidelines.
The award winner will receive a $500 cash prize, a commemorative plaque presented at the awards ceremony during the 2004 ASTD International Conference and Exposition, and a designated place on the conference program to present the research (with conference registration fee paid).
Submissions must be sent via email by September 22, 2003 to: Dr. John J. Sherlock, Assistant Professor of Human Resources, Western Carolina University, at sherlock@email.wcu.edu. For further information and submission guidelines, please contact Dr. Sherlock.
Performance Marketplace
Performance Marketplace is a convenient way to exchange information of interest to the performance improvement community. Take a few moments each month to scan the listings for important new events, publications, services, and employment opportunities. To post information for our readers, contact ISPI Director of Marketing, Keith Pew at keithp@ispi.org or 301.587.8570.
Books
and Reports
High
Impact Learning by Robert O. Brinkerhoff and Anne
M. Apking provides the conceptual framework for the HILS® approach
and is complete with integrated tools and methods that training
practitioners can use to help their organizations achieve increased
business results from learning investments.
ISD Revisited is a select collection of 56 articles from ISPI’s Performance Improvement journal focused ISD as practiced in the 21st Century. This compendium, with an introduction by Allison Rossett, provides a fresh perspective on ISD, presenting current thinking and best practices.
Conferences,
Seminars, and Workshops
Darryl
L. Sink & Associates, Inc. is offering the following workshops: Designing
Instruction for Web-Based Training, Dallas, September 22-24; Instructional
Developer Workshop, Chicago, September 15-17; Criterion-Referenced Testing
Workshop, Dallas, October 27-28. Visit www.dsink.com for
details and to register!
Add Performance and Pizzazz to Your Training. Whether it’s a 45-minute presentation or a week-long workshop, Thiagi can make your training come alive with interactive experiential activities. Nobody does instructional design faster, cheaper, or better than Thiagi. Visit www.thiagi.com today!
Consulting
Services
So you want to be a CPT? If you have the experience, but don’t
have the time, ProofPoint Systems has your solution. You provide the
information, and ProofPoint does the rest. Not sure what’s involved?
Call 650.559.9029, or email: info@proofpoint.net to
get started.
Job
and Career Resources
ISPI
Online CareerSite is your source for performance
improvement employment. Search listings and manage your
resume and job applications online.
Magazines,
Newsletters, and Journals
Chief
Learning Officer Magazine Let CLO deliver the experts
to you through Chief Learning Officer magazine, www.CLOmedia.com,
and the Chief Learning Officer Executive Briefings electronic
newsletter. Subscriptions are free to qualified professionals
residing in the United States.
Resource
Directories
ISPI
Online Buyers Guide offers resources for your performance
improvement, training, instructional design and organizational
development initiatives.
Training
Services
The
Power to Get Results. Martin Training Associates provides workshops,
services, and products that focus on developing hard and soft
skills in project management. Our methodology is universally
applicable to any project and project team type. Visit www.Martintraining.net for
details.
Newsletter Submission Guidelines
ISPI is looking for Human Performance Technology (HPT) articles (approximately 500 words and not previously published) for PerformanceXpress that bridge the gap from research to practice (please, no product or service promotion is permitted). Below are a few examples of the article formats that can be used:
In addition to the article, please include a short bio (2-3 lines) and a contact email address. All submissions should be sent to april@ispi.org. Each article will be reviewed by one of ISPIs on-staff HPT experts, and the author will be contacted if it is accepted for publication. If you have any further questions, please contact april@ispi.org.
PerformanceXpress is an ISPI member benefit designed to build community, stimulate discussion, and keep you informed of the Societys activities and events. This newsletter is published monthly and will be emailed to you at the beginning of each month.
If you have any questions or comments, please contact April Davis, ISPIs Senior Director of Publications, at april@ispi.org.
ISPI
1400 Spring Street, Suite 260
Silver Spring, MD 20910 USA
Phone: 1.301.587.8570
Fax: 1.301.587.8573
info@ispi.org
http://www.ispi.org