Student affairs programs and services are increasingly being called upon to show evidence that they are functioning effectively and that they positively impact the learning and development of college students.
Academic program reviews have a long history in higher education. Until the last 10 to 15 years, comprehensive program reviews of nonacademic areas have lagged behind those in academic areas because of the emphasis on accountability and the evaluation of higher education institutions’ instructional programs. According to the Research in Higher Education article “Developing and Implementing a Process for the Review of Nonacademic Units,” the academic support enterprise has generally been left alone other than issues of compliance with state regulations and policies.
Today, institutional effectiveness has become the focus in all aspects of higher education, University of Georgia professor emeritus Ted Miller wrote in a 2002 issue of NASPA NetResults. Student affairs programs and services “are increasingly being called upon to show evidence that they are functioning effectively and that they positively impact the learning and development of college students,” Miller said.
Since the Commission on the Future of Higher Education’s final report was released in 2006, regional accrediting agencies have been pressured to target evidence of student learning and the fulfillment of an institution’s mission as the focus of accrediting measures. Student affairs divisions are included in these expectations for such evidence.
Likewise, since its establishment in 1979, the Council for the Advancement of Standards in Higher Education (CAS) has moved from simply providing professional standards to facilitating the use of them for program self-study and evaluation. In the introduction to the 2006 “Book of Professional Standards in Higher Education,” then-CAS President Jan Arminio acknowledged the ever-increasing value the standards have produced for student affairs: “Recent research has indicated that CAS materials are used for guiding new programs, evaluating current programs, advocating for new initiatives, and guiding professional development. Since its inception, CAS has advocated for the value of self-assessment for its own merits. More recently, users of CAS standards have completed CAS self-studies in conjunction with or in preparation for an accreditation study.”
The nature of assessment and accountability in higher education in general and student affairs in particular has evolved today to the point where a more intentional strategy is necessary to meet multiple constituencies’ demands. A management framework is needed to help student affairs administrators and practitioners understand, analyze, and align their mission, programs, planning, personnel, processes, and resources. As Texas A&M University has discovered, comprehensive program reviews are valuable measures for evaluating the need.
The comprehensive program review process
The Division of Student Affairs at Texas A&M University has 17 departments that together provide quality service and learning environments for its students. To measure its success and areas for improvement, comprehensive program review has been a part of the division since 1997. The first iteration resulted in excessively large amounts of documentation on “what” departments were doing. In 2004–05, a pilot program, based on the principles of total quality management and the Texas Award for Excellence in Education, was conducted with two large departments within the division. The pilot resulted in the current review program for 2009–14. The focus is on “how” and “how well” departments are fulfilling their missions and aligning with the mission of the university and the Division of Student Affairs.
The comprehensive program review process has four parts:
- Preparation, which includes training the self-study team (three hours) and two six-hour workshops for the entire department
- The self-study, which includes writing the self-study report (one semester)
- The peer review process, which includes an off-site review of the self-study report and an on-site visit (two to three days)
- Action planning from the results of the self-study and peer review (due two months after the peer review team’s report is received)
At Texas A&M, the Office of the Vice President for Student Affairs leads the comprehensive review process. The Department of Student Life Studies facilitates the process as it trains, supports, and gives feedback to each department during the self-study and peer review. Student Life Studies pays each department’s peer review team expenses, including honorarium, transportation, housing, and food for the peer review team leader(s). Student Life Studies also assesses each department’s process—both self-study and peer review team—as well as the entire review program.
The components of the self-study include a concise departmental profile and 10 categories of study: program and services for students and stakeholders; human resources/staff; financial resources; facilities, equipment, technology; legal and ethical responsibility; assessment and evaluation; planning; leadership; outreach and engagement; and process management. The self-study structure is based on the standards provided by the Malcolm Baldrige National Quality Award Education Criteria for Performance Excellence and CAS. The process, in part, also is modeled on The Ohio State University’s program review.
Student Life Studies prepares and updates the Comprehensive Program Review Manual, which is given to each self-study team member at the inception of the department’s review process. It is a living document and is constantly being improved as each department adds the wisdom of its unique experience. The manual gives process and content direction to the user.
Creating a self-study team
The creation of the self-study team may be considered the most vital step of the comprehensive review process.These individuals will shape the department’s self-study document which is the foundation for the entire review. Three departments have completed the self-study and peer review process at Texas A&M: Disability Services, Multicultural Services, and Greek Life. Student Activities is currently engaged in the peer review process.
Creating a self-study team in a small to midsize department can be particularly challenging. The amount of time needed to work on the self-study, the strengths and weaknesses of the staff members, and other projects that they may be working on all have to be considered in developing the team. For Disability Services, it was easiest to create a small team with the director, assistant director (team leader), program coordinator, and student development specialist III. The self-study team members then each took responsibility for two to three categories in terms of writing the draft from the ideas generated during the brainstorming workshops with the whole department.
Multicultural Services also chose to look to the department’s leadership, which included the director, associate director, two assistant directors, and administrative assistant, as its self-study team. The rationale behind this decision was these individuals represented every part of the department and had the authority to determine the final answer for each of the self-study questions. In an effort to involve everyone in the department, staff members were assigned to at least one of the subcommittees, formed for each section of the self-study.
The Department of Student Activities is a large, complex organization with more than 40 staff members and 10 functional units and chose a different approach in selecting the self-study team. To promote support for the review process, the self-study team was intentionally designed to reflect the department. It was important to the department to have a team with an overall positive attitude, a commitment to a thorough and detailed process, and the ability to facilitate positive experiences for others involved. The self-study team includes members from each functional team from a cross-section of the department; all levels of positions are represented. Administrative and professional staff serve on the self-study team, as well as entry-level positions and members of the senior leadership team. The following factors were considered when determining the individuals to serve: attitude, position level in the organization, strengths (according to StrengthsFinder), ability to interact with others, and other
Preparation for a self-study
Preparation for a self-study is facilitated by the department of Student Life Studies. A two- to three-hour orientation is conducted for the self-study team. Then two six-hour workshops are conducted for the entire department before the self-study begins. (Departments do their best to involve all staff. Planning the timing of these workshops is one of the first challenges in initiating the comprehensive program review process). The workshops use focused brainstorming to develop answers to the questions posed. The goal is that at least 50 percent of the content of the self-study report be generated during these workshops.
The workshops were invaluable to the self-study process for Disability Services as every staff member attended and provided input on each category. The workshops provided information about the importance and value of the comprehensive program review process and helped staff who may not have been invested otherwise feel that there was a purpose in participating in the process. The majority of data used for the self-study report were generated in the workshops and then refined, expanded, and finalized during the writing process. Through the workshops, staff members understood the importance of comprehensive assessment, which enabled them to be supportive of the leadership team charged with writing the report.
The entire Multicultural Services department also participated in a two-day retreat designed to generate the beginnings of the answers to the self-study questions. Because everyone took part in this activity, the “voice” of the department was more inclusive than if just one individual had written each section alone. The subcommittee used the answers generated at the retreat to help shape responses for the assigned section.
Leading the self-study
The ability to organize and motivate are essential skills to demonstrate through the lengthy and at times tedious process. Developing a timeline for the six- to nine-month process will allow staff to understand the expectations and deadlines. Toward the end of the process when the peer review team schedule is being put together, there can be infinite details that need attention. Staying organized and keeping everyone motivated are critical, and self-study leaders should find ways to energize and reward those participating in the process. Rewards need not be expensive; for example, Multicultural Services gave a rotating award to recognize the individual who had made a significant contribution toward the process each week.
The self-study team leader for Disability Services was the assistant director. That individual’s responsibilities included scheduling leadership team meetings, setting deadlines, serving as a liaison with Student Life Studies for questions about the process, contacting the peer review team, and coordinating all aspects of the peer review team visit. The Disability Services director participated in the self-study process as an equal member of the leadership team. This worked well in Disability Services where the director and assistant director make up the departmental leadership team and are used to working together on projects with either one taking the lead.
The Student Activities self-study team is led by an associate director who is the departmental representative on the Division Assessment Team, and the director serves as a contributing member on the team. A nondepartmental staff member also is on the team to observe the process in preparation for her department’s review and to offer an outside perspective during the self-study team meetings.
The Student Activities self-study team received team-specific training prior to the comprehensive program review workshop, and each member is responsible for the writing and compilation of at least one category. A schedule of category completion was developed to guide the team’s progress. Team meetings were used to review information and then each functional unit representative brought the current category back to his or her unit for further feedback. The process was designed to be cyclical to maximize the involvement of the entire department throughout the process of writing the self-study report.
Preparing for a peer review team
Peer review teams at Texas A&M are led by a national or regional expert for the department under review. The department director generates a list of acceptable candidates, but the final choice of leadership comes from the Office of the Vice President. The balance of the team includes one undergraduate and one graduate student as well as a faculty member and a director from another Student Affairs department. Other members can be part of the peer review team due to the specific nature of the department (Greek Life had a former National Panhellenic Council president, for example), but keeping teams as small as possible makes for a more manageable and effective peer review team.
Choosing a peer review team can be somewhat challenging. Disability Services chose a director who happened to have worked in Disability Services in the past and has historical knowledge of the department. In identifying a faculty member or administrator to participate, Disability Services wanted individuals who had interacted with Disability Services, but not necessarily “fans” of the department. It was helpful that the faculty and staff members all had some knowledge of Disability Services. The team leader (outside member) was chosen based on experience with program review, reputation as an expert in the field, and experience in the profession.
The basic outline of the schedule was provided by Student Life Studies, and it was tailored to the constituents who are served by and work with Disability Services. The leadership team brainstormed who would be involved in the interviews and the team leader sent out invitations. The constituent groups for Disability Services included students with disabilities registered with the office, faculty, academic advisors, staff from other support units, and administrators. The site review team also met with the Disability Services staff in small group interviews.
As with many programs that student affairs professionals plan, beginning with the end in mind is a good philosophy to take when preparing the peer review team’s onsite schedule. Discuss which individuals or organizations should be invited to meet with the peer review team. Throughout the self-study, emphasize to staff that there are no wrong answers when they meet with the peer review team. Staff may be worried they will say something that might reflect poorly on the department, but it is important that the department leadership emphasize being honest throughout the process as the final report will not be helpful unless it accurately reflects what is happening in the department.
Action planning for the future
Peer review team reports are due from the team leader within four weeks of the on-site review. The department then has two months to combine its self-study and the peer review team report into an action or strategic plan for the next five to seven years. Action plans are presented and reviewed by the Office of the Vice President six months after initiation and then every year from that date until the next program review inception.
Following the Multicultural Services peer review team visit, department staff and especially the self-study team leader felt relief that the program review process was complete and they could turn their attention back to the department’s day-to-day operations; however, this would have been a mistake. Four to six weeks after the site review visit, the department received the peer review report and began developing action plans based on the findings.
For Multicultural Services, the comprehensive program review findings reinforced the direction the department’s leadership had initiated; however, a couple of topics surfaced that had not been specifically addressed prior to the process. The first was a focus on collaboration between student organizations housed in the department as well as collaboration between the department and other units on campus that planned diversity programs. While the student organizations and the department had always been open to co-programming and collaborating, the findings emphasized being strategic about these collaborations and not letting them occur by happenstance. To address this, the student organization orientation was modified to allow more opportunity for the organizations’ officers to spend time with one another and discuss events for the year. Advisors also have worked to communicate upcoming opportunities to the rest of the staff and to seek out collaborators in a more strategic way. A second finding was the need to review assessment information on a wider scale and not just look at it program by program. Department leadership now spends time at each meeting discussing common assessment questions and results for programs that may affect other department-sponsored programs. These are just two examples of how each finding should be critically examined to determine if or how it will be addressed.
Lessons learned or to be learned
If a university does not have a culture of comprehensive program reviews, a department desiring to undertake the process has several options. It might be easiest, as a beginning exercise, to use the CAS standards as a guideline for an internal review. Departmental leadership seeking a more evaluative and focused program review to include a peer review must gain the support of divisional leadership because significant time will be spent on the process. In an age of assessment and accountability, it should not be difficult to convince the appropriate individuals that conducting a comprehensive program review will result in information pertinent to budgeting and strategic planning for the coming years.
All review processes should have the flexibility and adaptability to fit the size, structure, and mission of the organization. In a small college setting or in a small student affairs division, it may be prudent to approach a program review with the complete division as the unit of review. Conversely, there may be large departments within a division that could profit from a review by individual subunits. For example, at Texas A&M University, the Office of the Dean of Students houses alcohol and drug education; student conflict resolution services; adult, off-campus, and graduate student services; student media; gender issues education; and new student programs. Therefore, it is important that the review aggregate the data provided by individual subunits for an overall evaluation and planning effort on behalf of the department. Separate peer review teams for each subunit would be cumbersome, time-intensive, and expensive, so a slightly larger team with several local or regional experts might be in order.
If funding for a comprehensive review process is not available, the peer review team could be composed of institutional members only. Also, it is not uncommon for peer review processes to include both an internal and an external review team. On the other hand, because a comprehensive review process only occurs every five years or more, it might be possible for departments to budget for the expense, use reserve funding, or find one-time funding sources.
A good source of feedback on the assessment process comes from the peer review team. At the end of each Texas A&M peer review team visit, the Student Life Studies director debriefs the peer review team after a closing exit interview with the department. Comments from the team have ranged from suggestions about sending the self-study report to them electronically (to be able to insert comments into the document itself) to oversights concerning individuals and/or groups they should have interviewed. In every case, these suggestions have been incorporated into the self-study manual and a future manual for the comprehensive review team leader. The team leader manual will include contracts for peer review team leaders, information on billing arrangements with Student Life Studies for peer review team expenses, suggested timelines, and the like.
A concluding reflection
Full staff participation and consensus are primary goals of a comprehensive program review process. Additionally, as D. David Ostroth reported in his 1996 New Directions for Student Services chapter “Comprehensive Program Review: Applying TQM Principles,” “The goals of the process were to be critical analysis, practical conclusions, and real change, not the production of lengthy reports.”
Benefits from increased engagement and collaboration toward these common goals are found throughout the process, not just in the end result. This is why, for the process to work and work well, each department needs to understand its role as the ultimate beneficiary of a program review regardless of how helpful it is to institutional accountability and accreditation.