By John T. Herrick and Roger B. Nielsen

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.

A Voluntary Faculty Development Program
Boise State University (BSU) serves nearly 20,000 enrollees in a metropolitan area in Boise, Idaho. Beginning in 2007, BSU’s Center for Teaching and Learning (CTL) began offering the Boise State Teaching Scholars (BSTS) program, which is designed to provide faculty with the opportunity to engage in yearlong group learning and share “best practice” teaching methods. The BSTS curriculum revolves around a group’s selected subject matter focus and includes common study materials, group projects that will benefit the university, and individual projects that will benefit the individual faculty participant or his or her students. Participation in this program is voluntary. Between 2007 and 2010, 54 faculty members participated in the program.

Meeting Program Objectives
In the fall of 2010, Susan Shadle, PhD, director of the CTL, asked for an evaluation of the BSTS program, and we, a two-member team of Boise State University graduate students, took on this evaluation project. The purpose of the evaluation was to determine if the goals of the program had been realized and if there were any opportunities for improvement. Stated objectives for the program included review and discussion of scholarly literature in the field of teaching, sharing best practices in teaching, research for group and individual projects, and community building between faculty from diverse disciplines.

Based on the following four high-level evaluation questions, we specified seven measurable dimensions with different levels of importance weighting, which are grouped into process and outcomes categories, as shown in Table 1:

  • How did the individual project, group project, facilitation, and meeting activities support the outcomes?
  • What insights did participants gain regarding the scholarship of teaching?
  • Did the program have a measurable impact on the teaching methods of the participants?
  • Did the participants develop and maintain cross-disciplinary relationships as a result of the program?

Table 1. Evaluation Dimensions and Importance Weighting

Process Outcomes
1. Individual projects (critical)
2. Group projects (desired)
3. Meeting activities and resources (important)
4. Facilitation (important)
5. Increased teaching scholarship (critical)
6. New or effective teaching techniques (critical)
7. Cross-disciplinary relationships (important)

Evaluation Approach and Data Collection
The primary intention of the evaluation was formative, to identify findings that would be used to further develop and improve the BSTS program. In answering whether the three outcome dimensions met their intended objectives, this evaluation project secondarily serves as a summative evaluation.

Theoretical Frameworks
We used Scriven’s (2007) Key Evaluation Checklist (KEC) as a framework that was specifically designed for application to program evaluation and systematically guides evaluators to make sure they include all important ingredients that will allow them to draw valid evaluative conclusions. Since it includes steps for evaluating both process and outcomes, it also helps avoid favoring one over the other. Additionally, we utilized Brinkerhoff’s (2006) Success Case Method (SCM) for the group project dimension because of its simple and effective way to evaluate the impact of training by evaluating individuals who applied the training successfully (Brinkerhoff, 2005). Finally, in developing recommendations for enhancing the quality of group projects, we used Keller’s (1987) ARCS Model of Motivational Design (as cited in Chyung, 2008, p. 56) because the participants needed to find value and expect results from the group project so they would be motivated to invest effort in it.

Data Collection
At the end of each year, the program conducted an exit survey with the group that finished the yearlong BSTS program. Our team reviewed and analyzed the extant data obtained from the exit surveys conducted between 2008 and 2010. Then, in the fall of 2010, we administered a follow-up survey to the same participants who, by that time, had one to three years to put into practice the skills that were gained through their participation in the BSTS program. The follow-up survey was used to both affirm and expand the data collected in the exit surveys. Of the 54 alumnus surveyed, 35 (64.8%) completed it. Additionally, we conducted two types of interviews. The first were interviews with BSTS staff, which helped establish insight into the need and intention of the program; and the second were interviews with BSTS participants, which provided further explanation of the survey data and gave us insight into how the program was successful on a personal level.

Main Findings
Our data showed that all seven dimensions were either good or excellent on a four-level scale (see Table 2). All three critical dimensions were rated as excellent, and we determined that the overall quality of the program was excellent. We concluded that the BSTS program, administered by the CTL, is a valuable asset to Boise State University’s academic community since it advances the scholarship of teaching. A majority of participants found the program intellectually challenging and socially supportive, positively affecting their careers.

Table 2. Summary of Evaluation Results

Dimension

Boise State Teaching Scholars

Weighting

Individual Project

X

Critical

Group Project

X

Desired

Meeting Activities & Resources

X

Important

Facilitation

X

Important

Increased Teaching Scholarship

X

Critical

New or Effective Teaching Techniques

X

Critical

Cross-Disciplinary Relationships

X

Important

Poor

Mediocre

Good

Excellent

Some of the strengths of the program are:

  • The participants found the individual projects helpful in improving their teaching.
  • The participants greatly valued the regular meetings for building professional relationships and integrating new teaching techniques in the classroom.
  • Participants become more aware that it is important to ground teaching in research-based best practices.
  • Participants incorporated new and more effective teaching techniques in teaching their classes as a result of participating in the BSTS program.

However, we observed that some participants showed a lower level of participation in the group projects, and we suspected that there might be a motivation issue around the group projects. For instance, we compared the responses from successful group project participants to the responses from less successful group project participants and found that the successful group members were involved in developing the mission of the group project and were highly aware of the purpose of their group project, while the less successful group members could not remember exactly what the group project was about. We therefore recommended that the program leaders apply Keller’s ARCS model to help the participants be more aware of the purpose of the group project and find it more relevant to their teaching.

References
Brinkerhoff, R. O. (2006). Telling training’s story: Evaluation made simple, credible, and effective. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Chyung, S. (2008). Foundations of instructional and performance technology. Amherst, MA: HRD Press, Inc.

Scriven, M. (2007). Key evaluation checklist. Retrieved from www.wmich.edu/evalctr/archive_checklists/kec_feb07.pdf

John T. HerrickAbout the Authors
John T. Herrick is currently employed by Blue Cross of Idaho as the manager of Information Systems Process Improvement and Security. He plans to complete his master’s in Instructional and Performance Technology in August 2011. He may be reached at jtherrick2@msn.com.

Roger B. NielsenRoger B. Nielsen is currently working as a graduate assistant and research assistant in the Instructional and Performance Technology department at Boise State University. He will complete his master’s degree in Instructional and Performance Technology from Boise State University in 2012. He may be reached at roger@nitecmfg.com.