By Shannon Rist and DeAnn Allen

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Instructional and Performance Technology department.

The Engineering Residential College
The Engineering Residential College (ERC) at Boise State University is a residential living-learning community that houses first- and second-year Engineering students. The ERC offers students access to resources such as programs, activities, study groups and tutoring, with the aim of enabling them to develop close networks with each other and professors, thus providing them with the support they need to continue in the Engineering program (Boise State University–University Housing, http://housing.boisestate.edu).

Evaluation
In Fall of 2010, Dr. Janet Callahan, the Faculty in Residence (FIR) of the ERC, requested an evaluation of the ERC program. A team of two Boise State University graduate students conducted a formative evaluation on the ERC program to assess how well the ERC was meeting its initial goals and determine what areas needed improvement. The main evaluation question was:

With the goal of fostering a sense of community and promoting academic achievement among engineering students, what aspects of the ERC are successful and what needs improvement?

Methods
To evaluate the overall value or quality of the ERC program, the team followed Scriven’s (2007) Key Evaluation Checklist. They began by conducting an interview with Dr. Callahan and gathering extant program data in order to determine the priorities of the program. After analyzing the data, they identified four program dimensions that would become the basis of the evaluation.

Next, the team needed to define the importance of each dimension in relation to its impact on the program (Davidson, 2005). To determine the importance, or weight, of each dimension, the team interviewed four current and previous ERC FIRs to draw on their knowledge of the program. The results of the interviews were compiled and the weighting determination was made based on equal consideration of the information provided by each FIR (see Table 1).

Table 1. ERC Evaluation Dimensions and Importance Weighting

Dimensions of Merit Weighting Reasoning
Development of Communication Network

Very Important

3 of 4 FIRs felt that a strong communication network is very important in helping students support each other and to know what resources are available to them.
Confidence in Communication

Very Important

3 of 4 FIRs felt that confidence in communication is a necessary aspect of the “I Can Do” spirit that is critical to the ERC program.
Sense of Community and Belonging

Very Important

1 FIR felt a sense of community and belonging is critical to the success of the ERC program, while 2 rated this dimension as very important and one important. This averages out to very important.
“I Can Do Spirit” 

(Self Assurance)

Critical

3 of 4 FIRs felt that an “I Can Do” attitude is crucial to the ERC program as it helps students develop and practice skills needed for success in engineering.

The team interviewed students of the ERC program, and conducted surveys with the students at the beginning (pre-survey) and end (post-survey) of the fall semester. The post-survey served as the primary data analyzed to draw evaluation conclusions. Both the pre-survey and interview data were used as secondary data to support, strengthen, and better understand the evaluation’s conclusions.

Surveys included questions on all four of the evaluation dimensions, and students’ answers to each question were measured on a 10-point Likert scale. The team evaluated the survey data against a 4-level evaluation rubric (Figure 1) to determine the quality of individual dimensions.

Figure 1
Figure 1. Scoring rubric for each dimension.

Findings and Recommendations
Based on the 4-level evaluation rubric, the evaluation team concluded that the quality of each dimension was Good; therefore, the overall quality of the ERC program was also Good (see Table 2).

Table 2. ERC Evaluation Dimensions with Weighting and Rating Scores

Dimension

Weighting

Rating

Overall Rating

Development of Communication Network Very Important

Good

Good

Confidence in Communication Very Important

Good

Sense of Community and Belonging Very Important

Good

“I Can Do” Spirit (Self- Assurance) Critical

Good

Based on interviews and qualitative data obtained from the open-ended questions in the survey, the evaluation team also identified the strengths of the program as well as some areas for improvement.

Strengths Overall, freshman students in the ERC program highly valued and felt a strong sense of community and support. They viewed Dr. Callahan and the Program Assistant as mentors and guides and felt that their weekly ERC class was effective in improving their leadership skills. Students felt that the ERC provided them with all the resources necessary to be successful engineers.

Areas for Improvement Students expressed a desire for increased training on online communication and more engineering-related activities in their weekly class. Also, although they enjoyed the social activities provided by the ERC, they requested more advanced notice and more structure to the events. Second-year students, who did not reside in the primary ERC lodging, did not feel the same sense of community as first-year students. However, toward the end of this evaluation project, steps had already been taken to create a residential housing unit specifically for second-year students from each of the five BSU residential housing programs.

The evaluation team presented the findings to Dr. Callahan and discussed ways to make changes, both short-term and long-term, in order to continue to facilitate the success of the ERC program at Boise State University. For example, to help students improve their “I Can Do” spirit (the critical dimension), the team recommended that the ERC incorporate more engineering-related activities in ERC events to help students increase their understanding of what an engineer does. To help students develop their communication network early, the team suggested scheduling trips during the first or second week of the semester instead of later in the course, and incorporating activities that also focus on students’ “I Can Do” spirit.

References
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.

Scriven, M. (2007). Key evaluation checklist. Retrieved from www.wmich.edu/evalctr/archive_checklists/kec_feb07.pdf

Authors
Shannon Rist is a Graduate Assistant in the Department of Instructional & Human Performance Technology. She will complete her master’s degree in Instructional & Performance Technology from Boise State University in 2012. She may be reached at shannonrist@gmail.com.

 

 

DeAnn Allen is a Graduate Assistant for the Department of Instructional & Performance Technology at Boise State University. She plans to graduate with a Master’s of Science in Instructional & Performance Technology in August, 2011, and go on to pursue a career in the field. She can be reached at DAMarie81@gmail.com.