the joys of rubric norming

At least twice a semester, I try (operative word) to run an informal rubric-norming session with all the instructors of our online credit course. The purpose of this is to determine what sort of papers “constitutes excellent, good, satisfactory, and poor papers” (Bean, 2011, p. 287).

Writing is often thought of as a subjective task, and evidence has shown that grading can be inconsistent in large courses where student work is graded by multiple teaching assistants. The purpose of norming a rubric is to try to improve both its validity and reliability.

While I don’t believe it’s necessary that our individual instructors need to grade the same way, I think the discussion helps us focus our priorities when it comes to grading these assignments. It can help move instructors beyond possibly shallow preferences as well as potentially help us clarify our expectations, perhaps even revise established outcomes or the rubric itself.

The quick and dirty steps that I’ve Frankensteined from various sources (including rubric superstar Megan Oakleaf) over the past year or so (though goodness knows that I usually run some sort of stripped-down variant on this):

  1. Model and offer a think aloud about how rubrics should be applied using student samples.
  2. Distribute copies of two papers and informally discuss the qualities that make one better than the other. They should be fairly similar in order to force instructors to discuss and articulate important nuances.
  3. The facilitator lists these qualities on the board to prime instructors; or to further match them to rubric dimensions; or has participants vote on the top traits. In the most recent session, I also wrote out the assignment outcomes on the board to help us stay focused.
  4. Distribute copies of another paper for instructors to grade on their own using the rubric.
  5. Individually report out ratings.
  6. When differences inevitably emerge, facilitator tries to dig into why one person awarded a D vs. the other who awarded an A on some dimension. The goal here is to reach a consensus, and it’s accomplished in a variety of ways. Someone might realize they’ve misunderstood the rubric, or maybe the group realizes that it’s a problem with how the rubric is worded.
  7. Repeat process from Step #3 with revised assignment and a new sample until there’s consistency in how everyone is grading.

That being said, I never actually get to that last step. This is a time-consuming process that allows us to only repeat the exercise a couple of times at the most each time. If we can at least get people to award similar grades (i.e. low A vs high B on the same dimension), then I will have considered that session a success.

Even if it never ends up being as robust a process as it should be, it gets better each time. The subsequent discussion is always very stimulating as we resolve our differences or learn about the unique mindsets that we bring to grading, and it is one of the things that helps make our group of instructors a true community of practice.



3 thoughts on “the joys of rubric norming

  1. Kate Herzog says:

    Brian, I’m a great believer in rubrics. And, while I agree that grading papers involves subjectivity and varies between individuals (and, sometimes I have found, between a single individual’s mind-set at the time of grading), I would propose that giving students the rubric at the time the assignment is originally put out and taking time to review the rubric with the class BEFORE they write their paper can be quite valuable. I’ve even asked my own students to grade themselves using the rubric and return it to me along with their paper.

    If you clearly tell them what you’re looking for, it makes the subsequent grading process easier AND also gives students the opportunity to compare their own expectations with yours.

    I also believe that having a shared rubric (or at least some consistent elements) among faculty (especially those in a given department or program) helps students develop their writing over time, rather than responding to varying expectations.

    1. bdleaf says:

      Thanks for the tips, Kate. Definitely can’t forget about the students in the process, and those are great strategies that I’ve admittedly applied inconsistently. One thing I’ve also wanted to try is having the class develop the rubric for a particular assignment. In the cases I’ve heard about, students ended up creating a more rigorous rubric than the instructor!

  2. Kate Herzog says:

    Brian, while I’d agree that involving the students in the development of rubrics certainly can get them actively engaged, I’d caution you to have your OWN criteria in your ‘back pocket’. There may be some things you ‘know’ must be present that the class might un/consciously choose to either ignore or overlook. This gives you a platform to explain the WHY [it’s important].

Leave a Reply

Your email address will not be published. Required fields are marked *