How Do I Design A Peer Review Program?

Tags:

Introduction
This section is intended to provide some guidelines for departments or units interested in instituting a peer review of teaching program. These guidelines are primarily in the form of questions, on which the department needs to come to consensus. The answers to these questions suggest which paths to peer review are likely to be most useful and productive for a department. Note that it is likely that different departments will arrive at somewhat different peer review programs because their perception of their needs and constraints differ. Because we think that the peer review program needs to address the interests of the reviewers and reviewees in a department, we view this spectrum of resulting programs as a healthy and positive outcome.

In summary, this section of the TLE site is intended to help departments design their own, unique, peer review program based on sound fundamental peer review ideas and the needs of the department. Once the program structure has been developed, departments might use other sections of this site to select specific peer review methods to implement which are consistent with the objectives of the department.

Background Work
Peer review of teaching can be a useful way to find out some things about student learning and faculty teaching. Effective implementation requires that an entire department discusses and, in some sense, agrees on why and how peer review is to be performed. It is particularly important that faculty who are likely to be reviewed have significant input into how and why such a review is to be performed. We think that peer review of teaching programs that are imposed on a department from outside the department or by a small subset of the department are unlikely to be effective or helpful. In this section, we pose a number of questions that we think a department should address as a group, and come to some consensus on answers. The questions are listed first, then discussed in a little more detail, including come common responses. The responses of your department may be similar to the range of responses indicated, or they may be quite different.

Questions:

  1. What kinds of information would the group like to get from a peer review process and what will they do with it?
  2. Who should get to see this information?
  3. Is peer review, as the group understands it, a good way to get this information?
  4. What other (non-peer review) information does the group need to make a complete picture?
  5. What are the major impediments to implementing a peer review process in this group?
  6. Is the group, in general, enthusiastic about using peer review for information gathering?
  7. How will the group organize itself to take the next steps toward implementation?

The intent of addressing these issues before beginning a peer review of teaching program is to clarify to all what is sought and what kinds of results are likely to be obtained. Among the many elements that contribute to successful peer review programs at those institutions that have been successful with peer review, is the sense of participation in the design and implementation of the program by all parties. Settling these issues beforehand can alleviate much of the concern (though not all) surrounding a peer review program, and make it more likely that the program will accomplish the goals set out for it.

We outline seven questions which should be considered in designing a peer review program. We suggest that a majority the discussion of these questions centers on question #1. This question, if fully explored and explicitly addressed, makes many of the other questions relatively easy to respond to.

Question Details

1. What kinds of information would the group like to get from a peer review process?

Common responses:

  • information on "How good is my teaching, and how could I make it better?"
  • information for promotion, i.e. a description of teaching and teaching development for an individual
  • information on what students have learned in a class/curriculum
  • information on relevance/interest/utility of course/curriculum

There are many more kinds of information which departments could seek from a peer review of teaching program. As a later questions suggests, there may be other better ways of finding some of the information in this list. This issue, of what kind of information does a department want is critical in designing a peer review of teaching program which will actually provide reliable information of the kind the department is seeking.

2. Who should get this information?

Common responses:

  • only the person being reviewed (may be appropriate when "How good is my teaching, and how could I make it better?" is the major departmental objective)
  • colleagues
  • department chair/administrators
  • divisional committee

This issue, who sees the results of a peer review, is often one of the most difficult in developing a peer review of teaching program. If a clear set of objectives for peer review has been established from question #1, there may be less contention on the issue of who sees the results. For personal review and feedback, for example, it may be quite appropriate that only the reviewer and reviewee see the results. For tenure promotion cases, evidence of peer review of teaching is now required by all UW-Madison divisional committees, so some documentation of the review will need to be presented to the department chair and the divisional committee. For other objectives developed by the department, discussion of who will see the results before any peer review has occurred is essential.

This issue gets at the nub of what will the results be used for. Will they be for promotion? Merit raises? Firing "lousy" teachers? Helping all faculty improve their teaching? As noted above, these issues should be settled before the program is initiated, and the decisions made need to be adhered to.

3. Is peer review, as the group understands it, the best way to get this information?

Common responses:

  • yes (reviewing peers is the best or only way to find out what we'd like to know)
  • no (reviewing peers really won't provide useful information on a topic)

While peer review can provide useful information on many teaching and learning issues, it does not provide information on all issues. In some cases, it may not be the best way to find out what a department wants to know. For example, peer review is a great way for a good teacher to find out how she/he could be better. It is probably not a great way to find out what students like best and least about a course or curriculum unless student interviews are included as part of the peer review. Peer review can help obtain information about teaching and learning which is not available by other means, but it should not be the only way in which information about teaching and learning is obtained. Different methods (student questionnaires, focus groups, other "tests") should be employed to obtain as complete a picture of the teaching and learning experience as possible.

4. What other (non-peer review) information does the group need to make a complete picture?

Common responses:

  • student questionnaires
  • student interviews
  • graduate interviews (5 or 10 years after graduation)
  • student performance measures (standardized tests, professional registration,…)

Except for student questionnaires, most of these alternative ways of gathering information are not well developed on most parts of the UW-Madison campus. Many departments are beginning to try some of these, particularly interviews of recent and 5 and 10 year graduates. The intent of this question is to focus on the objectives set out in question #1, and consider whether peer review of teaching can provide all or part of the information sought. If peer review can provide only a part of the information sought, what will constitute the other parts? (c.f. program assessment requirements for accreditation)

Reliance on student questionnaires, because of the well-developed process to administer them, is tempting. We encourage departments to think of student questionnaires as a source of a particular kind of information, and to include that information with other sources in evaluating teaching and learning.

5. What are the major impediments to implementing a peer review process in this group?

Common responses:

  • lack of time
  • lack of experience in peer review
  • mistrust of the process

For any department, there are real impediments to initiating a peer review program. If these impediments are strong enough, any peer review program may be doomed, and those attempting to develop one may be wasting their time. These issues need to be addressed at the beginning of the program, and they need to be dealt with. As each department character and culture is different, we doubt any canonical solution to these impediments exists. The members of the department are in the best position to decide what their major impediments will be, and how to overcome them. Some, like lack of experience, will go away as experience is gained. Others, like lack of time, won't go away and need to be addressed directly for a program to solidify. For example, "lack of time" usually implies that this task (peer review) is added to the range of tasks faculty are already responsible for, and must compete for time with other teaching and research activities. Clearly, this is true. Departments may shift this discussion somewhat by suggesting the issue is as much one of priorities (which is most important) as one of total tasks and total time. Departments have flexibility in establishing and promoting activities that they feel are of very high priority over others of a lower priority.

6. Is the group, in general, enthusiastic about using peer review for gathering information?

Common responses:

  • yes (we really want to get the information peer review can provide and are willing to do what it takes to get it)
  • no (we don't want to know this information badly enough to do peer review)

This is a question which summarizes, in some sense, several of the previous ones. At this point in a department discussion, the issues of what information might be gained from a peer review of teaching program and what it will cost in time and effort should begin to be evident. Departments need to decide whether the benefits outweigh the costs. If so, a peer review of teaching program could be quite successful; if not, it is unlikely that developing and implementing a program will be beneficial.

7. How will the group organize itself to take the next steps toward implementation?

Common responses:

  • we have no idea
  • let's look at the UW-Madison Peer Review of Teaching Web Site
  • forget those Bozos; let's go read lots of journal articles and invent our own peer review techniques

This is the key step towards implementation-what happens after a department has decided it wants to do peer review of teaching? Are people willing to put in the time and energy to make the program a success? Further, the department has agreed on what its objectives are for peer review, and what the results will be used for. What's next?

There are several issues included in "What's next?" For example, who will complete the description of the department's peer review program to formalize the "Why?, How?, Who?, When?" Who will seek out the resources available? The first issue needs to be settled by the department. We think that the broader range of ideas and opinions used to form the program, the better; so we would encourage the entire department to continue being involved in the development. As for resources, you're reading one of them: the UW-Madison Peer Review of Teaching Web Pages. You can use this site to help you select specific techniques your department might want to use and to help reviewers become familiar with their roles and responsibilities. You can also look for links to other sites with information on peer review of teaching.

If you want to create your own resources, feel free, but why not tell others about them so they can benefit from all your work?

Summary
There are probably as many kinds of peer review of teaching programs possible as there are departments at UW-Madison (or maybe more). A few select example programs are available for your review. The emphasis of this discussion of developing a peer review of teaching program is to avoid prescribing a perfect or model program which is unlikely to suit anybody, but rather to suggest a method by which departments could create their own program. This sounds like a lot more work than adopting someone else's tried and tested model program, but it's not. Most departments I know, would spend at least as much time arguing over which model to adopt and how to modify the one they chose as they would spend creating a program which is tailored to suit their specific department's needs. Spending that time "up front" seems slow and inefficient; it is likely to pay huge dividends later when the group doesn't need to debate small issues which arise, because they've already had extensive discussions about objectives and means, and have come to consensus on them. There will still be disagreements and discussions, but they are likely to be about the substance of the issues, rather than the range of preconceptions faculty bring to peer review. In the current vernacular, faculty are likely to all be "on the same page" if they have had discussions of objectives and means at the beginning.

Our group supports the idea that to overcome resistance to change, people involved in the change need to be involved in the development of the process from the beginning, and need to have a real voice in what will happen. "Change" in this case may be changing how some of the information regarding teaching and learning is gathered, or "change" may expand to changing how teachers and students perceive their roles in a learning environment.