by Jack J. Phillips

A previous article by Patricia Phillips focused on the benefits and barriers to implementing ROI, based on her new book, The Bottomline on ROI. This article focuses on the actual ROI process model, detailed in Measuring Return on Investment in Training and Performance Improvement Programs.

Building a comprehensive measurement and evaluation process is best represented as a puzzle in which the pieces are developed and put in place over time. Figure 1 depicts this puzzle and the pieces necessary to build a comprehensive measurement and evaluation process. The first piece of the puzzle is the selection of an evaluation framework, which is a categorization of data. The framework selected for the process presented here is a modification of Kirkpatrick’s four levels, to include a fifth level: return on investment (Kirkpatrick, 1975).

Figure 1. ROI: The Pieces of the Puzzle

Next, an ROI process model must be developed, showing how data are collected, processed, analyzed, and reported to various target audiences. This process model (see Figure 2) ensures that appropriate techniques and procedures are consistently utilized to address almost any situation.

Figure 2. ROI Process Model

The third piece of the puzzle is the development of operating standards. Labeled guiding principles, these standards help ensure the results of the study are stable and not influenced by the individual conducting the study. Guiding principles allow for replication so that if more than one individual evaluates a specific program, the results will be the same.

Next, appropriate attention must be given to implementation issues, as the ROI process becomes a routine process. Several issues must be addressed involving skills, communication, roles, responsibilities, plan, and strategies.

Finally, there must be successful case applications and practice describing the implementation of the process, the value that a comprehensive measurement and evaluation process brings to the organization, and the impact the specific program evaluated has on the organization.

The remainder of this article focuses on the individual pieces of the evaluation puzzle: developing a comprehensive ROI process.

Evaluation Planning
One of the most important and cost-saving steps in the ROI process is planning the evaluation. By considering several key issues during the planning, time, money, and frustration can be significantly reduced.

Collect Data
Following the planning process, implementation begins. Collecting both hard and soft data is central to the ROI process. Data are usually collected during two time frames. Reaction and learning data (Levels 1 and 2) are collected during the learning or performance improvement program. Application and business impact data (Levels 3 and 4) are collected after the program is implemented. A variety of methods are used to collect the post-program data to be used in the ROI evaluation, ranging from surveys to monitoring business records.

Isolate the Effects of the Program
In this step of the ROI process, specific strategies are explored that determine the amount of output performance directly related to the program. This step is essential because there are many factors that will influence performance data after a program is conducted. The specific techniques utilized at this step will pinpoint the amount of improvement directly related to the program. A few of these techniques include control groups, trend line analysis, or expert estimates.

Convert Data to Monetary Values
This step in the ROI model is absolutely necessary to determine the monetary benefits from a program. This step requires a value to be placed on each unit of data connected with the program. Approaches range from using standard values to expert estimates. The process is challenging, particularly with soft data, but can be methodically accomplished using the correct techniques.

Tabulate Costs of Program
The next step is tabulating the costs of the program. This involves monitoring or developing all of the related costs of the program targeted for the ROI calculation. The conservative approach is to include all of the costs—direct and indirect—so that the total is fully loaded.

Calculate the ROI
The return on investment is calculated using the program benefits and costs. The cost/benefit ratio (BCR) is the program benefits divided by cost. In formula form it is

The return on investment uses the net benefits divided by program costs. The net benefits are the program benefits minus the costs. In formula form, the ROI becomes

This is the same basic formula used in evaluating other investments where the ROI is traditionally reported as earnings divided by investment.

Identify Intangibles
In addition to tangible monetary benefits, most programs will have intangible non-monetary benefits. Data items identified that are not converted to monetary values are defined as intangible benefits.

Summary
The ROI process generates six types of data (reaction, learning, application, business impact, ROI, and intangibles), representing a balanced profile of success of a learning or performance improvement solution.

by Todd Packer

I’m honored to introduce I-Spy, a new feature of PerformanceXpress that will highlight relevant, interesting, and useful websites for Performance Technologists. In my work, I often use creative research techniques to uncover resources on the world wide web that can help individuals and organizations make effective decisions and achieve strategic goals. For this column, I hope to locate off-the-beaten path websites that can help you find similar thinkers, resources, work, new ideas, and sometimes just plain old fun.

Each month, I will highlight three sites connected by a general theme. While far from comprehensive, hopefully these sites will spark readers to look further and expand views about human performance technology (HPT). Please keep in mind that any listing is for informational purposes only and does not indicate an endorsement either by myself or the International Society for Performance Improvement (ISPI).

These are the general categories I will use for the sites featured:

The theme for our first column will be Academics. As a bridge between research and practice in HPT, ISPI connects Main Street, Wall Street, and the ivory tower. For a peek into the academic world, here are a few sites of note.

E-Klatch
The combination of research and education forms a basis of HPT and links to the field of educational research. The American Educational Research Association apparently shares ISPI’s interest in connecting process improvement, research, and practical application. Of particular note, given the Lessons in Leadership theme for the 2003 International Performance Improvement Conference & Expo, April 11-15, in Boston, MA, is the Call for Papers (November 12, 2002 deadline) for their journal Educational Evaluation and Policy Analysis for a special issue focusing on Education Leadership and Instructional Improvement. The American Educational Research Association (http://www.aera.net/) is concerned with improving the educational process by encouraging scholarly inquiry related to education and by promoting the dissemination and practical application of research results.

HPT@work
HPT practitioners often find satisfaction in academic work settings. The opportunity to pursue research to validate techniques, to teach and design curricula, to publish, and to apply the techniques within institutions draw many people to this area. Several websites and portals exist that link professionals to positions in academic settings. Keep in mind that HPT skills can also assist in the administration of academic settings, so reviewing positions in this area may be fruitful.

Here’s one:
Academic360.com is a meta-collection of Internet resources that have been gathered for the academic job hunter. It includes links to faculty, staff, and administrative announcements and is not restricted to teaching positions. For faster access the use of graphics and “bells and whistles” is kept to a minimum. Enjoy the site: http://www.academic360.com/!

I-Candy

Who says it’s hard to write like an academic? For an interesting approach to generating quick articles in “academic speak,” visit the Postmodernism Generator, which uses a computer application to randomly generate essays each time you refresh/reload the web page. This may come in handy for your next Performance Improvement article.

The Postmodernism Generator (http://www.elsewhere.org/cgi-bin/postmodern/) was written by Andrew C. Bulhak using the Dada Engine, a system for generating random text from recursive grammars, and modified slightly by Josh Larios.

NOTE: Listings above are for informational purposes only, and do not indicate an endorsement either by Todd Packer or ISPI.

by Carol Haig, CPT and Roger Addison, CPT

We spoke recently with Timm Esque, of Esque Consulting, about two emerging trends he thinks will create opportunities for HPT practitioners. Timm is the 2002 recipient of ISPI’s award for Distinguished Service and may be reached at tjesque@esqueconsulting.com.

Two Significant Trends
Organizations are beginning to shift from the short-term focus of the last 10-15 years to one centered on longer-term systemic management. The costs of basing strategic decisions on this quarter’s balance sheet are becoming painfully visible. Stakeholders, and particularly the shareholders among them, are losing patience as they experience the inevitable fallout from too much emphasis on the short term (e.g., unsustainable business practices and in some cases manipulative corporate bookkeeping).

The bookkeeping problems were predictable. The pressure for favorable business information is pushed from the board and executive office to middle management and the workers. As the pressure builds, people respond by presenting the information their superiors want to see or withholding unfavorable results. At one large organization, a department was three months behind in processing critical documents. Employees never reported this to senior management for fear of reprisal. When customers began complaining, the situation was discovered, resulting in a massive reorganization.

All that has changed recently is that some of this inaccurate information is making it all the way out to the stakeholders. This is partly due to the breakdown of checks and balances but also to the over-reliance on automated information systems. The second emerging trend is that managers are working around IT systems if they must, in order to get the information they need to make good business decisions.

Companies make huge capital investments in enterprise-wide systems. Far too often, these behemoths never do what management expected. Timm recently worked with a divisional vice-president who had been asking his finance department for accurate performance data for monthly review. He was repeatedly told that the new system would soon provide the information. Finally, the VP grew so tired of waiting that he told Finance he didn’t care if they had to use paper and pencil to calculate the figures, he wanted his general managers to see the data now and regularly in the future. The Finance people responded by creating some simple Excel-based tools to meet the VP’s requirements. This may seem like a step backwards, but not if that is what it takes to get the right information to decision makers.

Impact of These Trends
As illustrated in the above anecdote, the impact of these trends is that managers are getting serious (and pragmatic) about getting reliable information and using it to make forward looking business decisions. The recent popularity of buzzwords like “balanced scorecard” and “operational excellence” are further evidence that organizations are trying to move to a longer term, more systemic approach to management.

Implications for Performance Improvement
HPT is about systemic improvement. Timm believes the trend toward taking a longer strategic view and the focus on reliable performance data are an opportunity for HPT practitioners to move beyond short-term HPT solutions and help enable a very visible systemic approach. The balanced scorecard and operational excellence are fancy names for an effective performance system. Who knows more about performance systems than HPT practitioners?

One does not have to be a guru to enable systemic change in an organization. See Timm’s latest book, Making an Impact, for more information.

If you have any suggestions about trends driving performance in today’s business environment that you feel would be of interest to the PerformanceXpress readership, please contact Carol Haig at carolhaig@earthlink.net or Roger Addison at roger@ispi.org.

As a performance improvement professional, you are probably familiar with the HPT Model. For a quick review of the model, click here.

Play the HPT Model flash game by visiting: http://www.thiagi.com/ispi/sequence/html/TheHPTModel.html. You will find the seven steps of the HPT Model arranged in a random sequence. Rearrange the steps in the correct order by dragging and dropping them (before the timer counts down to zero). Remember, you can play this addictive game repeatedly. Each time you play, you will begin with a new random sequence of steps. You can even select a level to shorten the time limit to match your skills. Before you know it, you will become so fluent with the HPT process that everybody will be impressed.

Future issues of PerformanceXpress will feature new interactive games from Thiagi.

by Janet Wiscombe

Nelson’s study, conducted from September of 1999 to June of 2000, is based on responses from managers and their employees in 34 organizations ranging from Universal Studios to the U.S. Postal Service.

He says that several performance-related variables were found to have broad support from managers in the study, the majority of whom agreed or strongly agreed with the following items (listed with percentage of agreement):

Nelson also found that 72.9% of managers reported that they received the results they expected when they used non-monetary recognition either immediately or soon thereafter, and 98.8% said they thought they eventually would obtain the desired results.

Of the 598 employees who reported to the managers in the study, 77.6% said that it was very or extremely important to be recognized by their manager when they do good work. Employees expected recognition to occur: immediately (20%), soon thereafter (52.9%), or sometime later (18.8%).

“If you look at companies employees love to work for, you’ll find that they recognize their people and tell them they're doing a great job,” says Nelson, whose books include the best-selling 1,001 Ways to Reward Employees (Workman Publishing, 1994) and, most recently, Please Don't Just Do What I Tell You! (Hyperion, 2001). "Non-cash awards and incentives lower stress, absenteeism, and turnover, and raise morale, productivity, competitiveness, revenue, and profit."

Nelson’s mantra: “You get what you reward.”

NOTE: Janet Wiscombe is Associate Editor for Workforce. This article was reprinted with permission from Workforce, April 2002, p. 44.

In April 2002, a new element of the regular ISPI Annual Performance Improvement Conference & Expo program lineup was launched. The Performance Gallery is an interactive session, focusing on a visual display or single poster, each telling a story of a successful human performance technology (HPT) project and its results.

This year, the posters were rich and colorful, capturing the imagination and stimulating discussion. Although most participants reportedly stepped inside the room out of curiosity or for a quick look, most ended up engaged in a conversation with the authors of the posters. Among the participants was Eileen Banchoff discussing her model, Francois Lamott d’Incamps from France, and Connie Swartz sharing how to write a Request for Proposal (RFP) for instructional multimedia. The Performance Gallery became an opportunity for conference attendees from across the globe to share their experiences and knowledge in a comfortable, informal manner.

Take the opportunity now to submit a proposal to participate in the 2003 Performance Gallery. The benefits of submitting a proposal include:

Here are the guidelines to follow when creating your poster or visual aid:

You should also be aware that ISPI audiences are sensitive to commercial material. Logos may be used on title and summation graphics, but should not detract from the business message being presented. Advertising and sales promotions in any form are strictly prohibited.

The deadline for submission of your application to participate in the Performance Gallery at the 2003 International Performance Improvement Conference & Expo is October 14, 2002. For a copy of the application form and submission criteria, contact Ellen Bodalski Kaplan, ISPI Senior Director of Meetings, at ellen@ispi.org.

by Carol M. Panza, Director

The last copy of the International Society for Performance Improvement’s (ISPI) paper-based newsletter, News & Notes (November/December, 2001) contained an article that I contributed about the focus on the first “I” in ISPI which stands for International. Click here for a PDF of the full article. In that issue, I reported on the progress of a multinational group that was in the early forming stages and which was born out of the Chapter Leadership Workshop conducted by the Chapter Partnership Committee at the 39^th ISPI Annual Conference in San Francisco. Within the context of that forum, chapter leaders from existing and forming chapters from several European countries had an opportunity to get together and concluded that they could help each other and, potentially, accomplish things beyond any single country-based chapter by thinking more broadly. Well, think more broadly they did!

A forming/organizing meeting was held in early July in Brussels, hosted by MCE (Management Center Europe). Now, just short of one year later, the forming chapter ISPI, Europe, Middle East, Africa (EMEA) has just held a very successful conference in The Netherlands. There were more than 40 participants from 12 countries in attendance. (It was suggested that there were actually 13 countries represented, if we count Scotland as a separate country. But, we’re not going to go there!) The theme of the conference, appropriately enough, was Global Fluency, inspired by an article written by Don Tosti in the Performance Improvement journal (February, 1999). Don generously donated his time as one of the keynote speakers at the conference.

In my News & Notes article, I said, “There are many, many valuable professionals who currently belong to ISPI, and I’m not just referring to the Gurus! I’m talking about the folks you can make a call to or contact by email when you need some advice on a difficult issue. Or, it could be someone who can help you to confirm that you’re not crazy. You’re not alone out there.” ISPI EMEA actualized that concept in the design of the conference format. Concurrent sessions were selected based on their relationship to the global fluency theme and their interactive style. That is, the intent was to have facilitators leading each concurrent session, rather than the more traditional presenter format. During each session, one or more participants volunteered to summarize and “report out” on the learning from that session in a synthesis meeting at the end of the conference. The synthesis session was an opportunity for participants to learn a bit about sessions they were unable to attend. For facilitators, the synthesis session was useful feedback on what messages and learning were derived by participants who attended their sessions.

ISPI EMEA is already planning for its next conference in September, 2003 in the vicinity of Paris, France. The intent at this point is to keep the conference relatively small so that a highly interactive format can be maintained and, hopefully, expanded. This format clearly emphasizes the belief that we are ALL professionals with value-added contributions to make, not just the person or persons standing at the front of the room! Furthermore, as I said in my keynote address at dinner the evening before the conference sessions were scheduled, “Global fluency depends on global respect.” For anyone interested in viewing a PDF file of the slides used for my talk, click here. There is also a brief article that was distributed as a handout and is available as a PDF file by clicking here.

At the conference, I challenged our colleagues who participated to do several things to ensure that HPT (and ISPI) continues to develop its global fluency and international impact.

In fact, I would like to challenge everyone, not just the participants at the EMEA conference and the members of that forming chapter. All ISPI professionals need to contribute their ideas and experience if we as a Society are to continue to grow, not only in numbers, but also in the value that we offer to our current and potential membership and their clients.

by Carl Binder

by Aimee Boyd

Aimee Boyd is the evaluation specialist at E-Learning and Training Laboratories, IC² Institute, The University of Texas at Austin. She is a doctoral student in the department of Educational Psychology at the University of Texas at Austin, specializing in Quantitative Methods. She has an MA in Program Evaluation. Aimee may be reached at aimeeboyd@mail.utexas.edu.

by Brian Blecke, CADDI

• Myrene McLeod, Fisher & Paykel Appliances, New Zealand
• James Harbison, DST Output
• James Harris, SBC Southwestern Bell
• James Bonfiglio, Healthpoint
• Connie Perren, Equiva Trading Company
• R. Erin Brogan, US Coast Guard
• Indira Nath, Hewlett Packard
• Dean M. Hebert, Mindset for Performance

As I listened to my group and prodded them to reveal their insights, I learned two critical things about the participants. First, all of the members have strong roots in the training field and are personally and/or organizationally moving toward a more HPT-like model. Additionally, it was clear that training, in their varied roles, was still an important part of their day-to-day activities. Second, most of the participants in this group belonged to both ASTD and ISPI. This should not be a surprise to most of us in the Society (I, too, am a member of both). The richest part of the dialogue centered on, and sprang from, comparisons between the two organizations, and how the organizations interact with each other and their members. The focus group discussion provided a few key themes.

Theme 1
ISPI provides and promotes practical, research-based models, methods, and products. This group saw the research basis as a positive differentiator from other organizations. The members were unanimous in their praise. One member’s comment characterizes the group’s position, “[As a professional,] I need to be constantly refreshed and challenged, and the research is part of that.” Another said, “It is very important to base my choices and recommendations to my organization [on research found at ISPI]. It’s very important to be able to say, ‘It’s not just my idea, but I’ve researched it [at ISPI]’.”

The members roundly praised Performance Improvement as being a must-read on their evidently long list of reading materials.

Theme 2
ISPI should continue to strengthen its relationship with ASTD, in particular, and other societies, in general. From the conversation, I learned that the members perceived that ISPI and ASTD share a complementary, though sometimes overlapped, relationship in the marketplace. They saw a strengthened, differentiated relationship as valuable because it would reduce the amount of time required to “keep up to date.”

Theme 3
ISPI should continue to build through the local and international chapters. Quite a bit of time was spent discussing the process by which new members are acculturated into ISPI at the local and international levels. The members saw this as a critical opportunity for the growth of the Society’s membership ranks.

As the session drew to a close, more than one member expressed their happiness at being invited to participate in the focus group, and that this was a positive, first step in building on ISPI’s strengths and targeting areas for improvement.

The first focus group report for the Enterprise-Manager-Partner session appeared in the July issue of PerformanceXpress. Summaries of the other focus group sessions will continue to appear in future issues of the newsletter. If this summary, or the summaries of the other focus group sessions raise some ideas or comments, please make your opinion known by contacting the members of the Marketplace View 2002 Taskforce, Guy Wallace (guy.wallace@caddi.com) and John Swinney (jswinney@bandag.com).

by Barbara Gough, 2002 Conference Chair

Who picks the location of the conference?
The process of selecting a location begins with surveying members and other associations to select dates with the least number of conflicts. ISPI staff looks at many factors when determining when and where to hold a conference. A few of the major considerations include: where the conference was held in past years (West coast, East coast, etc.), accessibility, and price. These factors are weighted to help make the selection process easier. When the conference goes to a more popular city, known as a “first tier” city, it is usually held on less desirable dates to make the rates more affordable. When a “second tier” city, such as Cincinnati is selected, groups normally have more options with dates and less expensive hotel rates.

Why doesn’t ISPI provide lower-cost hotel options?
The ISPI conference is considered a mid-size event. That means that it is too small for most conference centers and too big for most hotels. This makes it challenging to find a suitable hotel to host the conference. When contracts with hotels are negotiated, ISPI must commit to a certain number of sleeping rooms in exchange for a certain number of meeting rooms to hold the concurrent sessions. Alternative hotels are only required when the Headquarter hotel sells all of its sleeping rooms.

Who selects conference educational sessions?
Conference sessions are selected by a group of people serving on the Conference Program Committee. This Committee is comprised of ISPI members who volunteer their time to evaluate all of the proposals received. Each proposal is reviewed and evaluated by three different individuals. Based on the criteria outlined in the Call for Proposals, all proposals receive a score. Proposals receiving the highest scores are accepted and included on the program.

Why do I have to pay to attend the conference if I am a presenter?
ISPI is a non-profit organization that relies on the full participation of its members. Presenting at the conference is considered a professional contribution and a way for our members to give back to their professional Society. In consideration of this contribution, ISPI offers one person designated as the “Lead presenter” a $50.00 discount off their registration fee.

How does the cost of ISPI’s conference compare to the cost of other conferences?
The ISPI conference is about 30% less expensive than other conferences. Other conferences that offer a paid registration to presenters have to recoup those costs by passing those costs on to paying attendees. ISPI’s registration policy, which requires that all individuals register for the conference, allows the Society to offer lower registration fees to all attendees.

Why is the cost of the banquet not included in the registration fee?
The banquet is a perfect opportunity for all members to network with old and new friends, celebrate a successful conference, recognize the Board of Directors, and honor Award of Excellence recipients. Over the years, hotel banquets have grown increasingly expensive. Instead of mandating that all attendees pay for an event they may not be able to attend, ISPI offers the banquet as an option, which helps to keep registration costs down for all attendees.

How do I show a return on the investment to my organization?
When you return home from a conference, there are many activities you can do to pass on the knowledge you gained by attending the conference. For example:

• Host a lunch and inform your team of the things you learned
• Call on the network you established at the conference to discuss issues you are having in your workplace
• Share with your team members the Conference on CD-ROM, including handouts and audio from most of the educational sessions

If we are an International Society, why aren’t flags from other countries displayed at the conference?
Normally, only the flags of the host country and host state are flown at the conference. During the 1999 joint ISPI & IFTDO Conference, flags from a number of different countries were displayed as a way to celebrate the joint effort of both organizations.

How can I get involved with ISPI?
There are many ways you can get involved in ISPI! Volunteer to help out on one of ISPI’s committees, support a local chapter, or even run for the Board of Directors. Visit www.ispi.org for a listing of committees, chapter groups, and Board nomination procedures.

If you have additional questions, suggestions, or comments on future ISPI conferences, contact Ellen Bodalski Kaplan, ISPI Senior Director of Meetings, at ellen@ispi.org.

The International Society for Performance Improvement (ISPI) would like to congratulate the list of professionals below who have taken advantage of the exemptions available during the grandparenting period and received the designation of Certified Performance Technologist (CPT) last month. Click here for a full list of CPTs. Visit www.certifiedpt.org, and apply today to receive your designation.

• Ronald J. Ryan, Amgen Inc, California, USA
• George W. Byars, Saudi Aramco, Dhahran, Saudi Arabia
• Nancy K. Luca, PRG-Schultz, Georgia, USA
• Burnes P. Hollyman, Austus Inc, Texas, USA
• Carol Haig, Carol Haig & Associates, California, USA
• Susan Coleman, Intelligent Decision Systems, Inc., Tennessee, USA
• Ellen Sue Menaker, Intelligent Decision Systems, Inc., Maryland, USA
• Lynn B. Kearny, Independent Consultant, California, USA

• Short “I wish I had thought of that” Articles
• Practical Application Articles
• The Application of HPT
• Success Stories

In addition to the article, please include a short bio (2-3 lines) and a contact email address. All submissions should be sent to april@ispi.org. Each article will be reviewed by one of ISPI’s on-staff HPT experts, and the author will be contacted if it is accepted for publication. If you have any further questions, please contact april@ispi.org.

Go to printer-friendly version of this issue.

Feel free to forward ISPI’s PerformanceXpress newsletter to your colleagues or anyone you think may benefit from the information. If you are reading someone else’s PerformanceXpress, send your complete contact information to april@ispi.org, and you will be added to the PerformanceXpress emailing list.

PerformanceXpress (formerly News & Notes and Quick Read) is an ISPI member benefit designed to build community, stimulate discussion, and keep you informed of the Society’s activities and events. This newsletter is published monthly and will be emailed to you at the beginning of each month.

If you have any questions or comments, please contact April Davis, ISPI’s Senior Director of Publications, at april@ispi.org.

ISPI
1400 Spring Street, Suite 260
Silver Spring, MD 20910 USA
Phone: 1.301.587.8570
Fax: 1.301.587.8573
info@ispi.org
http://www.ispi.org