Creating Rubrics

Red sharpie-type marker reading "Rubrics Guiding Graders: Good Point" with an A+ marked below

Red Rubric Marker

Instructors have many tasks to perform during the semester. Among those is grading, which can be subjective and unstructured. Time spent constructing grading rubrics while developing assignments benefits all parties involved with the course: students, teaching assistants and instructors alike. Sometimes referred to as a grading schema or matrix, a rubric is a tool for assessing student knowledge and providing constructive feedback. Rubrics are comprised of a list of skills or qualities students must demonstrate in completing an assignment, each with a rating criterion for evaluating the student’s performance. Rubrics bring clarity and consistency to the grading process and make grading more efficient.

Rubrics can be established for a variety of assignments such as essays, papers, lab observations, science posters, presentations, etc. Regardless of the discipline, every assignment contains elements that address an important skill or quality. The rubric helps bring focus to those elements and serves as a guide for consistent grading that can be used from year to year.

Whether used in a large survey course or a small upper-level seminar, rubrics benefit both students and instructors. The most obvious benefit is the production of a structured, consistent guideline for assigning grades. With clearly established criteria, there is less concern about subjective evaluation. Once created, a rubric can be used every time to normalize grading across sections or semesters. When the rubric for an assignment is shared with teaching assistants, it provides guidance on how to translate the instructor’s expectations for evaluating student submissions consistently. The rubric makes it easier for teaching assistants to give constructive feedback to students. In addition, the instructor can supply pre-constructed comments for uniformity in grading.

Some instructors supply copies of the grading rubric to their students so they can use it as a guide for completing their assignments. This can also reduce grade disputes. When discussing grades with students, a rubric acts as a reminder of important aspects of the assignment and how each are evaluated.

Below are basic elements of rubrics, with two types to consider.

I. Anatomy of a rubric

All rubrics have three elements: the objective, its criteria, and the evaluation scores.

Learning Objective
Before creating a rubric, it is important to determine learning objectives for the assignment. What you expect your students to learn will be the foundation for the criteria you establish for assessing their performance. As you are considering the criteria or writing the assignment, you may revise the learning objectives or adjust the significance of the objective within the assignment. This iteration can help you hone in on what is the most important aspect of the assignment, choose the appropriate criteria, and determine how to weigh the scoring.

Criteria
When writing the criteria (i.e., evaluation descriptors), start by describing the highest exemplary result for the objective, the lowest that is still acceptable for credit, and what would be considered unacceptable. You can express variations between the highest and the lowest if desired. Be concise by using explicit verbs that relate directly to the quality or skill that demonstrates student competency. There are lists of verbs associated with cognitive categories found in Bloom’s taxonomy (Knowledge, Comprehension, Application, Evaluation, Analysis, and Synthesis). These lists express the qualities and skills required to achieve knowledge, comprehension or critical thinking (Google “verbs for Bloom’s Taxonomy”).

Evaluation Score
The evaluation score for the criterion can use any schema as long as it is clear how it equates to a total grade. Keep in mind that the scores for objectives can be weighted differently so that you can emphasize the skills and qualities that have the most significance to the learning objectives.

II. Types of rubrics

There are two main types of rubrics: holistic (simplistic) and analytical (detailed).

Selecting your rubric type depends on how multi-faceted the tasks are and whether or not the skill requires a high degree of proficiency on the part of the student.

Holistic rubric
A holistic rubric contains broad objectives and lists evaluation scores, each with an overall criterion summary that encompasses multiple skills or qualities of the objective. This approach is more simplistic and relies on generalizations when writing the criteria.

The criterion descriptions can list the skills or qualities as separate bullets to make it easier for a grader to see what makes up an evaluation score. Below is an example of a holistic rubric for a simple writing assignment.

Table showing an example of a holistic rubric

Analytical rubric
An analytical rubric provides a list of detailed learning objectives, each with its own rating scheme that corresponds to a specific skill or quality to be evaluated using the criterion. Analytical rubrics provide scoring for individual aspects of a learning objective, but they usually require more time to create. When using analytical rubrics, it may be necessary to consider weighing the score using a different scoring scale or score multipliers for the learning objectives. Below is an example of an analytical rubric for a chemistry lab that uses multipliers.

Table showing an example of an analytical rubric

It is beneficial to view rubrics for similar courses to get an idea how others evaluate their course work. A keyword search for “grading rubrics” in a web search engine like Google will return many useful examples. Both Blackboard and Turnitin have tools for creating grading rubrics for a variety of course assignments.

Louise Pasternack
Teaching Professor, Chemistry, JHU

Louise Pasternack earned a Ph.D. in chemistry from Johns Hopkins. Prior to returning to JHU as a senior lecturer, Louise Pasternack was a research scientist at the Naval Research Laboratory. She has been teaching introductory chemistry laboratory at JHU since 2001 and has taught more than 7000 students with the help of more than 250 teaching assistants. She became a teaching professor at Hopkins in 2013.

Image sources: © 2014 Reid Sczerba

Quick Tips: Paperless Grading

Just in time for the end of semester assignment and exam grading marathon, The Innovative Instructor has some tips for making these tasks a bit less stressful.

Male instructor 's head between two stacks of papers.Last year we wrote about the GradeMark paperless grading system, a tool offered within Turnitin, the plagiarism detection software product used at JHU. The application is fully integrated with Blackboard, our learning management system. For assignments and assessments where you don’t wish to use Turnitin, Blackboard offers another grading option for online submissions. Recent updates to Blackboard’s include new features built into the assignment tool that allow instructors to easily make inline comments, highlight or strikeout text, and use drawing tools for freeform edits. All this without having to handle a single piece of paper.

If you don’t use Blackboard, don’t despair. The Innovative Instructor has solutions for you, too.  A recent post in one of our favorite blogs, the Chronicle of Higher Education’s Professor Hacker, titled Using iAnnotate as a Grading Tool, offers another resource. According to its creators, the iAnnotate app “turns your tablet into a world-class productivity tool for reading, marking up, and sharing PDFs, Word documents, PowerPoint files, and images.” This means that if you students submit documents in any of these formats (Professor Hacker suggests using DropBox, Sky Drive, Google Drive, or other cloud storage services for submission and return of assignments), you can grade them on your iPad using iAnnotate.

Erin E. Templeton, Anne Morrison Chapman Distinguished Professor of International Study and an associate professor of English at Converse College and author of the post, has this to say about how she uses iAnnotate’s features.

With iAnnotate, you can underline or highlight parts of the paper. I will often highlight typos, sentences that are unclear, or phrases that I find especially interesting. I can add comments to the highlight to explain why I’ve highlighted that particular word or phrase. You can also add comment boxes to make more general observations or ask questions, or if you would prefer, you can type directly on the document and adjust the font, size, and color to fit the available space.

I frequently use the stamp feature, which offers letters and numbers (I use these to indicate scores or letter grades), check marks, question marks, stars of various colors, smiley faces–even a skull and crossbones…. And if you’d rather, you can transform a word or phrase that you find yourself repeatedly tying onto the document into a stamp–I have added things like “yes and?” and “example?” to my collection. Finally, there is a pencil tool for those who want to write with either a stylus or a finger on the document.

Not an iDevice user? iAnnotate is available for Androids too, although it is limited at the time of this posting to reading and annotating PDF files.

The Professor Hacker post offers additional links and resources for paperless grading and more generally for those looking to move to a paperless course environment.  Be sure to read the comments for additional solutions.

Macie Hall, Senior Instructional Designer
Center for Educational Resources


Image Source: Microsoft Clip Art

To Curve or Not to Curve

A version of this post appeared in the print series of The Innovative Instructor.

Yellow traffic signs showing a bell curve and a stylized graph referencing criterion-referenced grading.Instructors choose grading schemes for a variety of reasons. Some may select a method that reflects the way they were assessed as students; others may follow the lead of a mentor or senior faculty member in their department. To curve or not to curve is a big question. Understanding the motivations behind and reasons for curving or not curving grades can help instructors select the most appropriate grading schemes for their courses.

Curving defines grades according to the distribution of student scores. Grades are determined after all student scores for the assignment or test are assigned. Often called norm-referenced grading, curving assigns grades to students based on their performance relative to the class as a whole. Criterion-referenced grading (i.e., not curving) assigns grades without this reference. The instructor determines the threshold for grades before the assignment is submitted or the test is taken. For example, a 92 could be defined as the base threshold for an A, regardless of how many students score above or below the threshold.

Choosing to curve grades or use a criterion referenced grading system can affect the culture of competition and/or the students’ sense of faculty fairness in a class. Curving grades provides a way to standardize grades. If a department rotates faculty responsibility for teaching a course (such as a large introductory science course), norm-referenced grading can ensure that the distribution of grades is comparable from year-to-year. A course with multiple graders, such as a science lab that uses a fleet of graduate students in the grading, may also employ a norm referencing technique to standardize grades across sections. In this case, standardization across multiple graders should begin with training the graders. Curving grades should not be a substitute for instructing multiple graders how to assign grades based on a pre-defined rubric (The Innovative Instructor: “Calibrating Multiple Graders”).

In addition to standardizing grades, norm-referenced grading can enable faculty to design more challenging assignments that differentiate top performers who score significantly above the mean. More challenging assignments can skew the grade distribution; norm-referenced grading can then minimize the impact on the majority of students whose scores will likely be lower.

A critique of curving grades is that some students, no matter how well they perform, will be assigned a lower grade than they feel they deserve. Shouldn’t all students have an equal chance to earn an A? For this reason, some instructors do not pre-determine the distribution of grades. The benefit of using a criterion-referenced grading scheme is that it minimizes the sense of competition among students because they are not competing for a limited number of A’s or B’s. Their absolute score, not relative performance, determines their grade.

There are multiple ways to curve grades.

Image showing a bell curve.I. The Bell Curve

Normalizes scores using a statistical technique to reshape the distribution into a bell curve. An instructor then assigns a grade (e.g., C+) to the middle (median) score and determines grade thresholds based on the distance of scores from this reference point. A spreadsheet application like Excel can be used to normalize scores. CER staff can assist instructors in normalizing scores.

Image showing clumping.II. Clumping

The instructor creates a distribution of the scores and identifies clusters of scores separated by breaks in the distribution, then uses these gaps as a threshold for assigning grades.

 

Image showing quota system.III. Quota Systems

Often used in law schools, the instructor pre-determines the number of students who can earn each grade. The instructor applies these quotas after rank ordering student scores.

 

Image showing criterion-reference grading.IV. Criterion-reference grading

Using a pre-determined scale, assessments are based on clearly defined learning objectives and grading rubrics so students know the instructor’s expectations for an A, B, C, etc.

 

During the 2011 Robert Resnick Lecture at Johns Hopkins, Carl Wieman, Nobel Laureate and Associate Director for Science at the President’s Office of Science and Technology, argued that most instructors are not trained to create valid assessments of student learning. Curving can be used as a tool to adjust grades on a poorly designed test, but consistent use of curving should not be a substitute for designing assessments that accurately assess what the instructor wants students to learn by the end of the course. CER staff are happy to talk to faculty about defining learning objectives and/or strategies for designing challenging and accurate student assessment instruments.

Additional Resources

• Campbell, C. (2012). Learning-centered grading practices. Leadership. 41(5), 30-33

• Jacobson, N. (2001). A method for normalizing students’ scores when employing multiple gradersACM SIGCSE Bulletin. 33(4), 35-38.

Joe Champion’s Grading Transformation Spreadsheet. This spreadsheet automatically curves students’ scores after the instructor copies the scores into the spreadsheet and sets a variable defining the amount of curve.

Michael J. Reese, Associate Director
Center for Educational Resources


Image Sources: © Reid Sczerba, 2013.

GradeMark Paperless Grading

GradeMark is a paperless grading system that gives instructors the ability to add comments and corrections to assignments submitted electronically. It is a tool offered within Turnitin, the plagiarism detection software product used at JHU. With its drag and drop functionality, among other features, GradeMark has the potential to save instructors a great deal of time when grading online assignments.  It is also easily integrated with Blackboard.

(Note: In order to use GradeMark, online assignments must be created using Turnitin. If using Turnitin within Blackboard, accounts are automatically created for instructors and students through the Blackboard system. If using Turnitin outside of Blackboard, the instructor is responsible for creating separate accounts for each student. Please click here for more information on Turnitin’s integration with Blackboard.)

Screen shot showing example of using GradeMark

GradeMark contains several different grading features:

  • Dragging and Dropping Quickmarks – Quickmarks are frequently used comments that are readily available to drag and drop into a student’s assignment. While viewing an assignment, the instructor can select from a panel of standard Quickmarks that come with GradeMark, or from a custom set that s/he has created.  For example, the abbreviation ‘Awk.’ is a Quickmark indicating an awkward phrase. The ability to drag and drop Quickmarks to an assignment, instead of typing them over and over again, can save instructors a lot of time.
  • General Comments – Each assignment has a generous space where general comments can be added.  General comments can be used to further clarify any Quickmarks that were added as well as discuss the assignment as a whole.
  • Voice Comments – A recent addition to GradeMark is the ability to add voice comments. A voice comment can be added to the assignment lasting up to three minutes in length.  An instructor can use the built-in microphone in his/her computer to easily record the message.
  • Rubrics – Rubrics created within GradeMark can help streamline the grading process by using a ‘scorecard’ approach. Specific criteria and scores are defined in a rubric that is then associated with an assignment. Instructors grade the assignment by filling in the scores based on the evaluative criteria in the rubric. There is also the option of associating Quickmarks with rubrics when they are added to the assignment.

Students are able to view their graded assignments when the ‘post date’ is reached. The post date is set by the instructor when setting up the assignment. Students have the option to print or save a copy of the graded assignment and can view only their own submissions.

GradeMark Logo showing grade book and apple

Advantages:

  • Flexibility in marking up assignments – Quickmarks, rubrics, text, voice comments all available.
  • Time saved dragging and dropping reusable comments.
  • Increased consistency in grading.
  • Clear feedback to students, instead of ‘scribbled margins.’
  • Opportunity to provide more detailed feedback to students including links and resources.
  • No need to download assignments – everything is web-based, stored online.
  • If the instructor is using Blackboard, when the assignment is graded the grade is automatically transferred and recorded into the Blackboard Grade Center.

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image sources: Amy Brusini screen shot of GradeMark example; GradeMark logo

Using a Rubric for Grading Assignments

Rubric comes from the Latin word rubrica meaning red chalk. In early medieval manuscripts the first letter of an important paragraph was often enlarged, painted in red, and called a rubric, leading to definitions of the term denoting the authority of what was written “under the rubric.” In the academic world the term has come to mean an authoritative rule or guide for assessment.

Instructor grading using a rubric

Most faculty, when preparing a graded assignment or exam, have expectations about how it should be completed, what will constitute a correct answer, or what will make the difference between an A and a C on a paper. Formalizing those thoughts into a written rubric – a template or checklist where those expectations are specified – has real advantages. First, it can save time when it comes to grading the assignment or test. Second, if you have Teaching Assistants, they will have a clear understanding of how to grade, and grading will be consistent across the sections. Third, it will make it easy to explain to students why they didn’t get that A they thought they deserved.

For a graded paper or project, it can be very helpful to share the rubric with the students when you give them the assignment. Seeing the rubric will help them to focus on what you feel is important. They will have a better understanding of the assignment and you will not only see better results, but have an easier time with the grading.

For more about creating rubrics see the CER’s Innovative Instructor print series article on Calibrating Multiple Graders: http://www.cer.jhu.edu/ii/InnovInstruct-BP_CalibratingGraders.pdf. For more on the reasons to use rubrics see: http://chronicle.com/blogs/profhacker/is-it-too-early-to-think-about-grading/22660.

Macie Hall, Senior Instructional Designer
Center for Educational Resources


Image source: Microsoft Clip Art, edited by Macie Hall