PerformanceXpress

International Society of Performance Improvement Newsletter
October 2004

EPSS: A 20-Year Retrospective
by Kim E. Ruyle, PhD

• Database of job-required information that is organized to facilitate rapid access and optimize clarity
• Calculators and wizards that simplify and automate procedures
• Decision-support modules that provide intelligent assistance with problem solving
• Embedded tutorials and simulations that provide instruction in work-related concepts and procedures

Just as a hand-tool leverages physical capabilities, an EPSS leverages cognitive capabilities. An EPSS can provide adaptive support for a full range of cognitive tasks—it makes performers smarter!

1. Web-based EPSS applications assist sales reps throughout our 10-step sales process. These applications support research and analysis, account qualification, establishing design requirements, evaluation, account management, and even best-practice sharing. The online support is augmented with laminated job aids, e-learning modules, and some limited instructor-led training.
2. Web-based EPSS applications support our engineers and project managers in much the same way, addressing a wide variety of tasks such as product application, project planning, engineering layout, and system commissioning. All of these applications are available directly through intranet access, but we’ve also incorporated many of them in a large-scale, scenario-based e-learning application.

Kim E. Ruyle, PhD, is the Director of Learning and Development for Siemens Logistics and Assembly Systems. Kim is a long-time ISPI member and has contributed to publications and many conferences. He may be reached at kim.ruyle@siemens.com.

Advertisement  Would you like to advertise in this space? Contact marketing@ispi.org

Trendspotters: Future Watch, Featuring Bob Horn
by Carol Haig, CPT and Roger Addison, EdD, CPT

Bob Horn, Visiting Scholar at Stanford University, Distinguished Consulting Faculty at Saybrook Graduate School and Research Center, and Chair of Information Mapping, Inc., is also the recipient of ISPI’s 2004 Thomas F. Gilbert Distinguished Professional Achievement award. In his current work, Bob, who may be reached at hornbob@earthlink.net, is using his Information Mural Visualization Method to help with the analysis and communication of intricate public policy issues. He talked with us about three predictions that will affect all of us in the workplace in the near future.

Top Three Predictions
First, Bob predicts the development of an international auxiliary language for business and commerce that will evolve from a rapid increase in the use of visual language in our communications. Fluency in this new language will become a core communication skill requirement in the workplace.

Second, today’s available content management software is robust. The proliferation of knowledge management efforts, the identification, organization, and storage of learning objects, and related needs for collection and arranging information for reuse will continue to grow in organizations. This trend will encourage the use of a structured information architecture to collect and organize information.

Third, there is a growing recognition among leaders and theorists that organizations are “social messes.” These are “problems about which different people have very different perceptions and values concerning their natures, causes, boundaries, and their solutions.”

Reasons for These Predictions
Visual language is widely used today, from advertising to the ubiquitous PowerPoint^® presentation. In some companies, creating an informal PowerPoint “deck” to describe an idea and emailing it to colleagues is a standard method of communication. And we have the famous death-by-PowerPoint presentation that is so much a part of sharing information in groups.

People are ready for a better way to present and exchange ideas, and they are seeing the possibilities in visual language. They are learning what it is that words convey best and what visuals do best. (For more on this topic, click here to read the interview with Lynn Kearny, CPT in the April issue of PX.) There is constantly improving software available, millions of pieces of clipart to choose from, and best of all, we don’t have to be able to draw.

As we, in our organizations and in society today, grapple with ever more complex problems and ideas, we are becoming more aware of the value of the tight integration of words and visuals (such as diagrammatic elements and images) and their power to clarify intricate issues. “You can’t handle complexity in organizations without visual language,” Bob says. And, he explains that there are global advantages to visual language: the use of pictures “disambiguates” words, and vice versa.

If we look around us, we can see that organizations have been struggling with information management for some time. Now, senior leaders have identified the need for storing information in an architectural structure for reference and reuse. The globalization of commerce means that the architecture must be uniform. It must use content management software with established standards that let users identify small chunks of information to assemble into a document, for example. The Institute for Electronic and Electrical Engineers (IEEE) has taken the lead and established standards for reusable learning objects. Soon companies will know exactly how much of their stored information is reused.

It is likely that most of us can identify a number of social messes once we understand what they are. Many government and social programs such as those for mental health services, long-term care, and health care in general qualify. Familiar to most of us are the dysfunctional organizations in which we work, and we could probably have a contest to choose the one with the biggest social mess. Many long-term predicaments, such as nuclear waste disposal, are also social messes with stakeholders worldwide and complicated communications surrounding the issues.

Bob has created mess maps, large diagrams that represent the many points of view that comprise social messes. The maps help locate the problems expressed and show where they intersect. Mess maps are great tools for task forces that must form mental models of the problems they tackle. Having a common mess map can help these groups take action quickly.

How Organizations Will Be Different
As organizations experience the failure of various large-scale implementations, they will develop increased humility. At the same time, the pain of these defeats will create an openness to learning how to manage broad, complex issues more effectively. The truth is that there is a lack of expertise in unraveling the issues of large-scale social change.

With the development of knowledge management architectures, executives responsible for enormous websites will be better able to manage stored information. Their organizations will become more adept at solving problems by locating information quickly and reusing it appropriately.

If organizations fail to manage gaps and the white space (see Improving Performance: Managing the White Space on the Organization Chart by Rummler & Brache), they will be managing more social messes.

Implications for HPT
As performance improvement practitioners, these predictions present opportunities for us to make significant contributions in the organizations we serve. We can mitigate messes by ensuring that we stay true to the foundations of HPT:

• Take a systematic approach to all problems and opportunities
• Start with the end in mind
• Link to business requirements
• Create partnerships
• Add value for the organization

Using our tools, we can make a difference in our organizations and in the world. (For more on this topic, click here to read the interview with Margo Murray, CPT in the March 2002 issue of PX.)

If you have any predictions about the future of HPT that you feel would be of interest to the PerformanceXpress readership, please contact Carol Haig, CPT, at carolhaig@earthlink.net or http://home.mindspring.com/~carolhaig or Roger Addison, EdD, CPT, at roger@ispi.org.

Beyond ROI: Managing Performance Systems as Business Assets
by Diane Gayeski, PhD

As performance consultants, we are increasingly under the gun to demonstrate the return on investment for our work. While it’s clear that most of the value of an organization comes from intangible assets like employee knowledge and motivation, reputation, and management processes and culture, there are still no established ways to either measure or maximize them. We have a great opportunity here since we are the managers of these important strategic assets.

At the same time, our field has been trying to implement ROI processes, with marginal success because:

1. It’s impossible to separate out the effects of various performance improvement interventions from other factors that impact individual behaviors and business performance. Therefore, most measures of ROI are not credible.
2. Most powerful interventions are not discrete projects with a specific goal and end, but rather infrastructure assets that undergird a variety of ongoing uses.
3. ROI calculations are overly simplistic and don’t factor in important variables such as the impact of taxes and the cost of capital. Therefore, when HPT professionals present ROI data to executives and financial professionals, they are not taken seriously.

Over the past six years, I’ve been researching and teaching new methods for managing performance systems. Reading the literature in finance, working as a visiting scholar at the Hanken School of Business and Economics in Helsinki, and applying my new ideas in my consulting work has allowed me to develop and refine some new approaches that are really exciting. Let’s just take one short example:

Let’s say that you created a new online system for field repair technicians; the application allows them to capture and share best practices and to enroll in short e-learning modules. The system cost $200k to implement, and within its first year it resulted in about $50k of efficiencies. However, you expect that as the knowledge base builds, you’ll be able to gain $100k per year in cost savings so you think the system will pay for itself in a few years. The ROI isn’t so simple to measure, because you’ve got to look at important financial factors such as the cost of capital, depreciation, and ongoing maintenance costs for the system. But let’s put aside those issues for the moment.

While $100k may look like a lot of money to you, it’s probably lunch money to your CEO. However, here’s a more powerful way to position this: The knowledge system is an infrastructure asset. If it can be positioned as producing a relatively permanent cost savings, you can actually calculate its impact on stockholder value.

Your pre-tax savings is $100k, but after taxes, that comes out to be only about $64k. HOWEVER, let’s say that your company’s Price/Earning ratio is 24.1 (that’s close to the typical S&P 500 average). What you’ve really done is to increase shareholder value by more than $1.5 million (see table below)!

You’re already a hero, and now you’re talking in REAL dollars. But there’s more. If you’re thinking like an asset manager, you may also be able to “productize” your intervention by copyrighting materials or even patenting certain processes. Not only does this protect your investment, but it could also provide the framework for you to market your successful solution or license some of its underlying algorithms, therefore creating a revenue stream. You should also consider ways to publicize your intervention by writing articles, making conference presentations, and getting mentioned in publications that are influential to your organization’s customers and partners. This helps to strengthen the brand. Are you seeing new ways to become a more entrepreneurial asset manager?

If we want to truly become business partners, we need to:

1. Speak the language of business (and this means understanding at least some basic financial measures and concepts)
2. Re-position ourselves from project producers to being managers of strategic infrastructure assets
3. Demonstrate and document how we impact important measures such as brand strength and intellectual capital, and get those measures included in published scorecards, annual reports, etc.
4. Create ways for training and HPT to become profit centers through protecting and leveraging an organization’s intellectual property

NOTE: This article is based on Diane Gayeski’s new book, Managing Learning and Communication Systems as Business Assets published by Prentice-Hall.

Diane Gayeski is CEO of Gayeski Analytics and maintains academic appointments as Professor in the Park School of Communications at Ithaca College and Adjunct Professor at Boise State University. She recently earned the Excellence in Scholarship award from Ithaca College for her ongoing research contributions to the field of learning and communications. Diane may be reached at diane@dgayeski.com.

Advertisement  Would you like to advertise in this space? Contact marketing@ispi.org

Performance-Based Instructional Systems Design: What's That?
by Sivasailam “Thiagi” Thiagarajan, PhD, CPT

Did you attend ISPI’s 2004 Performance-Based Instructional Systems Design Conference in Chicago? Well, even if you didn’t, here’s a contest just for you.

When I told a friend that I was making a presentation on performance-based instructional systems design, she exclaimed, “What’s that?” She also mumbled something about the fact that I always talk funny. I wasn’t sure whether she was referring to my accent or the tendency among ISPIers to create terms that sound like the title of someone’s doctoral dissertation. Anyhow, I tried to explain what performance-based instructional systems design is all about. And I failed miserably. That gave me an idea for this contest: Can you define performance-based instructional systems design?

I have an international panel of judges ready to objectively evaluate the definitions and select the best one. If your entry is selected, you win! (What exactly do you win? Fifteen seconds of fame. We will publish your name in a future issue of PX.)

You may be thinking, “But I don’t know anything about performance-based instructional systems design!” Don’t let that stop you. After all, as a PX reader, you should be able to figure it out.

Important Rule
Your definition should be exactly 16 words long. No more, no fewer. (Don’t ask “Why?” Please humor me for the present. It will all become clear at a later date.)

More Rules

• Email your contest entry to contest@thiagi.com.
• Judging criteria include accuracy, creativity, and appeal to PX readers.
• Include your name and email address with your entry.
• You may send more than one entry (but you can win only once).
• The decision of our judges is final.
• Results will be announced in a future issue of PX.
• All entries will be posted to a website (of course, you will get full credit).

The deadline for the contest is 11:59 pm (EST), October 15, 2004. All entries must be received by the deadline.

From the Board:
Dispelling the Myth: The Top 10 Questions You Have About Certification
by Andrea K. Moore, CPT, ISPI Director

So you’ve got some questions about certification. Maybe you’re thinking about taking the plunge. Maybe you’re already a Certified Performance Technologist (CPT) and are wondering if there is anything else you need to do. In any case, read on. Hopefully, you’ll find all the answers to your certification questions below.

10. How many CPTs are there?
As of September 1, 2004, there were approximately 1,000 CPTs. Here is the yearly breakdown of CPTs joining since the program started in 2002:

There were so many CPT applications in 2003 because of the “grandparenting” clause.

9. What types of companies have CPTs on staff?
The companies that have CPTs are diverse. There are banks, universities, storage companies, computer companies, printer companies, hospitality organizations, medical and pharmaceutical companies, and government agencies that have multiple CPTs on staff. Here is a list of the five companies that have the most CPTs in their organization:

To view the full list of current CPTs, click here.

8. Are people that are hiring actually looking for CPTs?
If you look at the CPT website, you’ll find a section called, Comments from CPTs. These are comments from those who have their CPT designation. You’ll notice that most of them say that the CPT allows them to open doors with clients and internal customers to speak about performance improvement using a systematic process. Others mention that the CPT signifies professionalism, rigorous standards, and sound principles, which allow the designees greater opportunities within their organizations.

Recently, I did a search on Monster.com for performance technology. There were 25 hits and most of them had references to preferring a candidate who was certified. One company in particular stated, “Possession of a Certificate in Human Performance Technology preferred.” Another company requested in their desired qualifications, “Professional certification relevant to Learning, Quality, or Performance Technology.” So, you can see that having the CPT designation is becoming a way for candidates to distinguish themselves from other applicants.

7. I turned in my CPT application in July. Why haven’t I heard anything?
No matter when you turn in your CPT application, there are two times a year that ISPI processes CPT applications, June and November. It usually takes 2-4 weeks to process an application so you should receive notice in July or December.

6. Once I’m certified, do I get a certificate?
No. The CPT designation is based on performance, not class attendance and a test. If you want a piece of paper, however, you may feel better when you receive the 16” x 18” framed CPT recognition plaque.

5. Does certification last a lifetime?
No. To keep your certification current, you must submit re-certification forms every three years. The purpose of these forms is to show how you have made strides to stay abreast of the field of human performance technology.

4. Do I fill out the same forms for re-certification as I did for my CPT?
No. The original certification forms were based on performance in project work. Now that you’re a CPT, you need to show how you have stayed current through continuing education, professional development, and volunteer service in our field. To review the re-certification requirements, click here. As an example, you can earn 12 re-certification points by attending ISPI’s 43rd International Performance Improvement Conference in 2005. You’ll need to earn a total of 40 points to re-certify your CPT designation. Keep in mind that for re-certification, you are once again required to sign the Code of Ethics.

3. How will I know it’s time to re-certify?
The time period for re-certification is every three years. That means that 10% of you will need to re-certify in 2005. CPTs who received their designation in 2003 will re-certify in 2006, and the remaining CPTs from 2004 will have until 2007. No matter when your original certification date was, ISPI will send you an email reminder before your time is up. That way you’ll have plenty of time to attend a conference or institute, or volunteer for an organization to get re-certification credit.

2. How can I find out if I’ve already earned re-certification points?
To review your member information, visit ISPI’s website: www.ispi.org. Once there, log onto My ISPI and click on your profile. To see which ISPI conferences you’ve attended, committees you’ve worked on, and presentations you’ve submitted, click on Activities. Then, you can transfer that information to the re-certification forms and total up the appropriate number of points.

If you are active in your local chapter, ask them to provide you with a list of the meetings you’ve attended and the volunteer positions you’ve held. You’ll want to make sure to get credit for these types of activities that International ISPI does not track.

1. What is the best thing about being a CPT?
Some may say it’s the glamour, others may say it’s the fame, but nothing quite measures up to the pride you feel when you become a CPT. Pride in collaborating, pride in taking a systems view, pride in showing value, and pride in focusing on results. It’s these four principles that you abide by in the work you’ve done in the past, the work you do today, and work you will do in the future.

Get Certified and Stay Certified! For more information, visit www.certifiedpt.org.

It Keeps Getting Better: Things You Should Know About Vancouver

New in 2005! Explore Canada’s West Coast
ISPI invites you to journey with us as we explore Canada’s West Coast during ISPI’s 43rd Annual International Performance Improvement Conference, April 10-15, 2005, in Vancouver, British Columbia, Canada. The experience begins the moment you enter this surreal world through fresh evergreen trees surrounded by the aroma of a British Columbia Rainforest. As you make your way into the rainforest, wander into Okanagan Wine Country or visit Little Italy for delectable pasta. If it’s sushi you’re craving, plan to spend time in the Pan Asian Square where food and culture come together. Sample seafood and listen to crashing waves as you watch a beautiful sunset off the shores of Vancouver Island. Whatever path you follow on your journey, plan to enjoy food, drink, camaraderie, and an energetic, electric ambiance that will enrapture you for hours. 

One ticket for the Thursday evening networking event is included in all full conference registration categories. Additional tickets may be purchased for spouses, guests, or local clients.

Bring-a-Colleague Offer
With the overwhelmingly positive reaction to ISPI’s Annual Conference, it’s no surprise that half of our 2004 attendees in Tampa were repeat participants. And, one-quarter had attended five or more ISPI conferences!

In Vancouver, we would like to introduce, and re-introduce, more of your colleagues to ISPI, by offering the Bring-a-Colleague rate once again. When you register for the full conference at the member or delegate rate, you may also register a colleague for only $450—provided your colleague has not attended an ISPI Annual Conference in the past three years (2002-2004).

Travel to Canada
Travel between the U.S. and the Canadian border now requires a passport or original birth certificate. ISPI has compiled a list of helpful resources to help facilitate the process of obtaining or renewing a passport for individuals who plan to attend the conference. Find out the requirements for travel to the Vancouver, instructions on how to obtain a U.S. Passport, Visas, Customs, and more by clicking here. It is also important that you confirm the acceptable travel documents with your air carrier when making your airline reservations.

Conference Updates
You’ll also want to check out the latest conference updates, including CPT Events and Workshops. In addition, don’t forget the deadline to submit your Award of Excellence entry is October 15, 2004. For more information, click here.

Back to School: Orchestrating Teamwork
by Sheila Scanlon Wilkins, CPT

In September 2003, I digressed from my consulting practice to teach elementary school, making a return to my earliest career as a music teacher. It was an enlightening experience to return to the public school classroom after many years as both a corporate employee and an independent performance improvement consultant.

I found myself asking questions I would ask any corporate client:

• What results do we expect from our educational system?
• What competencies should students possess upon graduation?
• Are students fully prepared to enter the workplace and make an immediate contribution?
• Are the expectations we have of our educational system realistic?

My assignment was to teach beginning flute, clarinet, alto sax, trumpet, and trombone classes. I had between five and ten students in each class. Spending a year teaching children music taught me that putting together a concert-quality band is very similar to assembling an award-winning team.

Goal Setting
If you don’t know where you’re going, anywhere will do. Trite, but very true! Effective leadership is critical for achieving results in the corporation and in the classroom.

At the outset of each class, the students and I set a year-long strategic goal:

• By the end of the school year, you will demonstrate your proficiency in playing the flute, clarinet, alto sax, trumpet, or trombone.

Goals need to be translated into everyday language so people understand what it is they are expected to do. Translated, this particular goal meant:

• You’ll be able to play the primary basic notes on your chosen instrument, play a concert B flat scale and arpeggio accurately and musically.

It was important to state the goal precisely and gain agreement from everyone.

Selecting Metrics
Next, we decided how we would measure our progress toward that goal. Some metrics were already established:

• Winter concert
• District festival
• Spring concert

Each music program presented a different repertoire—incremental in difficulty and sophistication. Our quality metrics included:

• Playing the correct notes consistently
• Watching the conductor (me) fastidiously
• Listening to each other regularly
• Observing the tempo markings and dynamic levels accurately

A long-term goal is often easier stated than accomplished. Achieving results requires focused, consistent effort and a commitment to hang in there when the going gets tough.

Determining Action Steps
Most goals require several intermediate objectives with specific action steps. Learning to play an instrument can be a challenge. Getting started involves basic and incrementally difficult steps:

• Taking care of a musical instrument
• Assembling the instrument; disassembling it
• Producing a sound on an instrument without squeaks
• Learning the fingering and positions for each note
• Learning to read notes
• Moving from one note to another to several notes in a musical phrase
• Learning the language of music

Teamwork
By mid-October, we were operational; that is, students knew how to care for their instruments properly, produce reasonably good tones, and play three to four notes accurately. Preparing to perform was the next step. That involved teamwork. A music team is like a work team.

• The group must have a purpose for working together.
  • A musical piece requires several instruments playing different parts simultaneously.
• Group members must be interdependent; that is, they need each other’s skills and commitment to arrive at mutual goals.
  • A musical selection does not sound whole if some of the instrumental parts are missing.
• The group must be accountable as a functioning unit within a larger organizational context.
  • Each family of instruments must play its part of the piece accurately.
• The group members must coordinate laterally.
  • Not all instruments play every note, play the same notes, or play at the same time.
• Significant face-to-face interaction among group members is required.
  • Each instrumentalist must be competent at his/her individual part and each group of instruments must be in balance with the others.
• There are no individual stars; rather, there is a team of star performers.
  • Any team, even a musical group, is only as good as its members.

Reaching Goals
As we prepared for each concert, all the required skills came into play. Beginning instruments, now a beginning band, readily achieved their stated goals for each concert because students were clear at the outset what they needed to do. They also realized it took teamwork to get there.

For the final concert of the year, the band chose to play Highlights from Harry Potter, a challenging piece even at the middle school level. With lots of practice, individually and collectively as a band, they played the selection extremely well. It sounded like Harry Potter! Imagine the sense of personal satisfaction and team exhilaration at being able to stretch and achieve a goal beyond their expectations. It was heady!

Summary

• No matter what the situation, leaders must provide direction, set performance expectations, remove obstacles to getting the job done, and inspire and motivate their followers.
• Teams are more productive when members individually have a high commitment to deliver performance results.
• Teams have more fun, and having fun is integral to successful performance.

Sheila Scanlon Wilkins, CPT, a long-time member of ISPI and principal of The Wilkins Group, works with clients in various industries, helping them achieve their targeted business goals. She is an Arts Commissioner for the City of Walnut Creek, CA. Sheila also works in the local public schools teaching instrumental and vocal music. You may reach her at Sheila@wilkinsgroup.com.

CPT Application Deadline Approaches

The Certified Performance Technologist (CPT) designation is awarded by ISPI to experienced practitioners in the field of performance improvement and related fields such as instructional design and organizational development whose work meets the 10 Standards of Performance Technology and other application requirements. The deadline for submitting your application to become a CPT must be received at ISPI by November 15, 2004, or it will be held until the next processing deadline of June 15, 2005. Visit www.certifiedpt.org for more information on becoming a CPT and to download the application.

Measurement Counts! Using High-Fidelity Simulations to Certify Performance
by Carl Binder, PhD, CPT

In past issues, this column has mostly addressed nitty-gritty measurement topics about how to decide what to measure, and how to measure in ways that will optimally support management and instructional decisions. The focus has been mainly quantitative. With that as background, this month’s column is a bit of a digression—but an important one, I think.

Measurement During Training and Beyond
As regular readers must realize, when my associates and I measure the results of training or other performance improvement interventions, we look for things to count. We identify accomplishments (job or process outputs), sub-accomplishments (sometimes called milestones) or types of behavior to count, and we graph repeated measures to monitor changes in level, trend, variability (bounce) or quality over time (per minute, per hour, per day, per week, per month). We use the “performance pictures” formed by charted data to decide whether or not we’re achieving desired results and whether we need to change our approach.

As you might also be aware, we use counts of responses or outputs per minute on practice exercises and tests to give trainers, coaches, and learners themselves feedback about whether and how rapidly they’re achieving desired component-level performance goals, and whether they need to make changes in their learning or practice strategies to optimize progress. We count and time behavior to determine if and when learners achieve fluency. With these measurement procedures in place, traditional percent-correct testing or rating scales became largely irrelevant because they are so far less sensitive than count/time performance measures.

Using Simulations to Measure Application
Toward the end of training or coaching programs, or after completion of modules or units intended to develop a particular repertoire or type of performance, we often use performance tests in the form of high-fidelity simulations to determine whether and how well individuals can perform. If, as we do, you distinguish among three stages of learning—1) initial learning, 2) practice for fluency, and 3) application—performance tests focus on stage 3 in which learners combine fluent components to produce important job outputs.

Our distinguished colleague, Judith Hale (2000), has written an excellent book about how to design and implement performance-based certification programs in organizations. Covering virtually every aspect and angle related to certification—including the business drivers and details to consider when implementing such programs—she touches on the use of performance tests as part of an overall picture.

In such diverse areas as sales, customer service, equipment maintenance, and accounting, we’ve found it possible to combine real or simulated inputs, tools, and other elements of real-world situations into test experiences that challenge performers to meet actual on-the-job requirements. In sales, for example, we’ve designed case study scenarios in which trainees review telephone transcripts, account notes, financial records, industry background, and other information and then use sales reference materials and collateral to prepare for and execute simulated sales calls with peers or managers. Insurance company representatives (Enrollers) have used job aids to customize standard presentations for specified audiences, delivered the presentations to peers, and responded to tough questions and objections. Maintenance technicians have been given equipment with specific symptoms to diagnose and repair, and accounting trainees have been asked to complete portions of audit procedures with pre-determined errors to detect, etc. In each case, evaluators use behavioral checklists to monitor critical features of required performances and evaluate job outputs. In some cases, performers must meet specific time requirements in order to pass.

In our experience, most such performance tests are pass/fail rather than numerically evaluated. When trainees are unable to perform to criterion, trainers or managers give feedback and direction for practicing missed components in preparation for another attempt, often with a different scenario or case.

When my associates and I hear the term “certification,” we usually think of this type of high-fidelity simulation or case-study challenge. We realize that many of our colleagues, including Dr. Hale, use a broader set of evaluation methods, including percent-correct knowledge tests, for what they call certification. But when we propose to “certify” that individuals or groups are definitely able to perform important tasks or jobs, we use the most realistic and representative scenarios we can arrange so that performers, their trainers, coaches, and managers can see without additional interpretation whether and how well they are actually able to perform.

I’d be most interested in hearing from readers about experiences using simulations as performance tests, and the results achieved with that approach.

Reference
Hale, J. (2000). Performance-based certification: How to design a valid, defensible, cost-effective program. San Francisco: Jossey-Bass/Pfeiffer.

Dr. Carl Binder is a Senior Partner at Binder Riha Associates, a consulting firm that teaches clients the FluencyBuilding^™ performance improvement methodology, practical measurement strategies and tactics, and Six Boxes™ performance management. His easy-to-remember email address is CarlBinder@aol.com, and you can read other articles by him at www.Binder-Riha.com/publications.htm. See past issues of this column by clicking on the “Back Issues” link at the bottom of the blue navigation bar to the left.

Performance Technology Makeovers
by Allison Rossett, PhD, CPT, Column Editor

• Surveyed training or organization design professionals at about half of our primary manufacturing units (concentrated on facilities that were considered to be high performers) and key business leaders (corporate level)
  • Questionnaires sent out by email (18 open-ended questions)
  • Questionnaires were forwarded by respondents to two additional sites that were not “high performers”
  • 80% return rate from site contacts
• Compiled and summarized survey data and sent to all respondents
• Sent executive summary of findings to each site manager, mill training contacts at each site, and corporate manufacturing leaders
  • Asked for confirmation of findings. Did the findings ring true? How would they expand upon what we’d learned in the surveys? Received very little additional input, but no disagreement.
• Formed an advisory group to:
  • Prioritize gaps
  • Identify resources to close gaps
  • Assess existing courses, identify vendors
  • Write work plan and budget to adapt existing courses, develop new courses, and deliver

• While we did some data gathering, our analysis could have been improved by including input from:
  • Members of target audience (front-line leaders themselves), including high performers and a random sample drawn from all front-line supervisors
  • Data from lower performing mills
• A key assumption was not rigorously validated. We needed more specific information on the “new” front-line leaders
  • Demographics of front-line leaders who are not ready to retire and those soon to be promoted to leader positions
    • Are they, in fact, really less technically knowledgeable?
    • What and how much do they need to know? Could the technical expertise reside elsewhere?
    • What are specific ways that front-line leader technical expertise can improve operating unit performance?
  • The role of the first-line leader was changing in many locations—some were high-performing work teams, some traditional work systems, and variations in between. We did not have a good way to address this.
• We had a few systemic questions on our survey related to organizational barriers, succession planning, and knowledge management but did not use that data to generate a tailored solutions system.
  • Our group was committed to training as the primary method to enhance the contributions of front-line leaders to bring about improved operating reliability and efficiencies
  • We had hoped to publish a list of key technical competencies with resources, self tests, and possibly set up a certification requirement, but did not vigorously pursue this
  • We did not provide any other support for front-line leaders such as online references, technical contact lists, online community, etc.
• When offering supervisory training (most of which was classroom courses provided by high-quality vendors), many mills opted to send operators in addition to, or instead of, first-line supervisors. This created problems for the instructors and lowered the value for the target audience in some cases.
  • This was partly due to our strategy to offer regional or mill-based classes to minimize time away from the mills. But in some cases, the mills couldn’t free up enough supervisors to fill a class.
  • Another issue was that our project looked at front-line leaders’ technical competency in isolation. We did not look at the technical competency of the entire mill system to see if there were other segments with more critical technical training needs.

Sherry Ryan is an instructional designer at Weyerhaeuser Company in Tacoma, WA. She is looking forward to completing a Masters degree in Educational Technology through San Diego State University’s online program in May 2005. Sherry may be reached at sherry.ryan@weyerhaeuser.com.

Seeking Applicants for Editor-in-Chief of Performance Improvement

The International Society for Performance Improvement (ISPI) is seeking an ISPI member who has the flexibility to take on the commitment and responsibilities of Editor for Performance Improvement (PI).

We’re looking for a member who can demonstrate an extensive knowledge of human performance technology (HPT), has a professional HPT network, and possesses an editorial review ability. The Editor will be responsible for acquiring, reviewing, and selecting manuscripts and will contribute suggestions and ideas toward the editorial direction. The Editor will work with authors and potential authors to maintain the highest standard of editorial content and will work directly with ISPI’s Senior Director of Publications, who is responsible for all production and distribution. The Editor reports to the Executive Director, who serves as Publisher of Performance Improvement. The position requires a two-year commitment, commencing in April 2005. The Editor will receive $10,000 a year as compensation for the invested time and effort.

PI is published 10 times a year and is distributed to more than 5,000 members, subscribers, and institutions. For an application and instructions, or for questions regarding the position or the application process, please contact April Davis, ISPI Senior Director of Publications, by phone: 301.587.8570 x112; by fax: 301.587.8573; or by email, april@ispi.org.

Newsletter Submission Guidelines

ISPI is looking for Human Performance Technology (HPT) articles (approximately 500 words and not previously published) for PerformanceXpress that bridge the gap from research to practice (please, no product or service promotion is permitted). Below are a few examples of the article formats that can be used:

In addition to the article, please include a short bio (2-3 lines) and a contact email address. All submissions should be sent to april@ispi.org. Each article will be reviewed by one of ISPI’s on-staff HPT experts, and the author will be contacted if it is accepted for publication. If you have any further questions, please contact april@ispi.org.

PerformanceXpress is an ISPI member benefit designed to build community, stimulate discussion, and keep you informed of the Society’s activities and events. This newsletter is published monthly and will be emailed to you at the beginning of each month.

If you have any questions or comments, please contact April Davis, ISPI’s Senior Director of Publications, at april@ispi.org.

ISPI
1400 Spring Street, Suite 260
Silver Spring, MD 20910 USA
Phone: 1.301.587.8570
Fax: 1.301.587.8573
info@ispi.org
http://www.ispi.org