By Stacey Olachea, Colleen Olson, and Ben Davis

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.

Evaluation Request
In the fall of 2013, Total Vision Soccer Club (TVSC; a pseudonym), a competitive youth soccer club, requested an evaluation of their College Advisory Program (CAP) to verify whether the program aids players in finding the best college fit based on their needs and to determine key areas of improvement. Through the game of soccer, TVSC offers a variety of opportunities for youth players, ages 2-19, with a focus on having fun, developing skills, and providing players with competitive playing opportunities. As the competitive soccer business expands, clubs such as TVSC continue to identify opportunities that enhance their competitive advantage. CAP is one such program, designed to support TVSC’s athletes in pursuit of their college goals by helping them learn how to navigate the college recruiting process, market themselves to college coaches, and increase their exposure to potential colleges and universities. A team of students from Boise State University’s Organizational Performance and Workplace Learning graduate program conducted the evaluation study.

Methodology
This evaluation was conducted as a formative evaluation to support TVSC’s desire to use the conclusions to verify if the existing program achieves its intended outcomes and to identify critical areas for program improvements. The evaluation team used Scriven’s key evaluation checklist (Davidson, 2005) to guide the evaluation methodology. To understand the various if-then relationships between the means and end results of the program, the evaluation team designed a program logic model (W.K. Kellogg Foundation, 2004) using input from two TVSC key stakeholders, the girls’ director of coaching and the boys’ director of coaching (see Figure 1).

Figure 1. TVSC’s program logic model.

Tales-Fig1

The team then focused on the activities, outputs, and outcomes of the logic model to help identify the significant program dimensions. After reviewing the team’s understanding of the dimensions with the client, together they determined that the evaluation should focus on two areas of process and three areas of outcomes for five dimensions of merit (see Table 1).

Table 1.  TVSC Dimensions of Merit

Dimensions

Dimensional Categories

Dimensional Evaluation Questions

Program design

Process

How well is the CAP designed (to provide services and tools) to aid players/parents in the college recruitment process?
Marketing design How well is the program marketed to eligible players/parents?
Staff knowledge

Outcome

How accurately are the TVSC coaches presenting the CAP and the CAP services to players/parents?
Player/parent knowledge, skills, and abilities How prepared are the players/parents to navigate the collegiate recruitment process?
College placement rates To what extent do the players using CAP have a higher success rate in obtaining college opportunities?

 

The team and client collaboratively identified data collection methods and data sources to be used to investigate individual dimensions. This robust process, which included data collected from various TVSC stakeholders and program recipients using multiple methods, enabled the team to effectively triangulate the results. The results from the surveys, interviews (individual and focus groups), and extant data reviews were tabulated using a four-level rubric– poor, mediocre, good, and excellent. During data collection, the team determined there was insufficient data to effectively evaluate the college placement rates dimension. This dimension was removed from the overall results and conclusions. The team analyzed data for the remaining four dimensions to identify the overall effectiveness of the program and then prioritized recommendations for improvement.

Conclusions
Based on the data collection results, the team concluded that the overall program rating was mediocre to poor. This rating was based on the evaluation of four dimensions according to their levels of importance (see Table 2).

Table 2. TVSC CAP Dimensions, Merit Determination, and Weighting

Dimension

College Advisory Program

Weighting

Poor

Mediocre

Good

Excellent

Program design

Extremely Important

Marketing design

Important

Staff knowledge

Very Important

Player/parent knowledge, skills, and abilities

Extremely Important

 

The client identified the level of importance of each dimension. Along with helping assess the overall performance of the program, these importance weightings also assisted the team in prioritizing final conclusions and recommendations for improvement. The team paid significant attention to the two dimensions that were identified as extremely important.

Recommendations
Although the evaluation revealed several strengths, the team also identified several opportunities to improve the program. Collected data revealed an eagerness by TVSC and its staff to help players achieve their collegiate goals. Conversely, the collected data also identified frustration among the club’s primary audience (players and parents) in successfully supporting the players in their pursuits. To support TVSC’s desire to identify program improvements, the evaluation team identified several recommendations for improvement that were aligned with the club’s priorities as well as those that would have the greatest potential impact.

The team’s highest priority recommendations focused on enhancing the program’s design and repositioning and expanding information, tools, and services to enhance players’ and parents’ knowledge, skills, and abilities to navigate the college process successfully. Secondary recommendations focused on enhancing TVSC’s marketing efforts; providing training for coaches on CAP’s goals, objectives, and resources; and implementing tracking of college placement. In summary, the evaluation team presented TVSC with the following recommendations:

  • Refocus the program’s design to emphasize individual advising sessions
  • Provide “how to” learning tools and materials
  • Align program content and materials with age-specific information
  • Brand all CAP materials to support players’ and parents’ ability to identify resources
  • Create a comprehensive workbook tool for players and parents
  • Provide a CAP training sessions for coaching staff
  • Enhance the TVSC website to strengthen program marketing and player exposure
  • Track and share college placement results

Furthermore, as an overarching recommendation the evaluation team concluded it would be beneficial for TVSC to review the objectives and purpose of the CAP against TVSC’s overall mission and goals. Conducting a high-level review of this nature would further assist TVSC to prioritize recommendations.

Post-Evaluation Results
The evaluation report and recommendations were presented to the TVSC director of girls’ coaching and the TVSC director of boys’ coaching, who, in turn, have presented the findings to the board of directors and the club’s president and technical director. The board of directors is currently reviewing the report, but the initial feedback has been positive and they are excited to begin implementing several recommendations.

References
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.

W. K. Kellogg Foundation. (2004, January). Logic model development guide. Retrieved from http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx.

About the Authors

StaceyOlachea Stacey Olachea has worked in the financial, health, and government fields for the last 15 years applying her knowledge of human resource practices to enhance overall work environments and production. She is currently a graduate student in the Organizational Performance and Workplace Learning program at Boise State University. She will complete her master’s degree in May 2014, and may be reached at staceyolachea@u.boisestate.edu.
ColleenOlson Colleen Olson has over 18 years of experience in the learning and performance field working in a variety of business sectors with employees across all organizational levels. She is currently a graduate student in the Organizational Performance and Workplace Learning program at Boise State University. She will complete her master’s degree in December 2014, and may be reached at colleenolson@u.boisestate.edu.
BenDavis Ben Davis has over 10 years of experience in learning and development within the financial services industry, with an educational background in marketing, graphic design and learning, technology. He is currently in his last semester as a graduate student within Boise State’s Organizational Performance and Workplace Learning program. He may be contacted at ben@battleborn.us.