Lunch and Learn: Creating Rubrics and Calibrating Multiple Graders

Logo for Lunch and Learn program showing the words Lunch and Learn in orange with a fork above and a pen below the lettering. Faculty Conversations on Teaching at the bottom.On Friday, December 15, the Center for Educational Resources (CER) hosted the second Lunch and Learn—Faculty Conversations on Teaching—for the 2017-2018 academic year.  Laura Foster, Academic Advisor, Public Health Studies, and Reid Mumford, Instructional Resource Advisor, Physics & Astronomy, presented on Creating Rubrics and Calibrating Multiple Graders.

Laura Foster led by giving us a demonstration of her use of Blackboard for creating rubrics. She noted that she might be “preaching to the choir” but hoped that those present might take back these best practices to their colleagues. Noting that many faculty have negative opinions of Blackboard, she put in a plug for its organizational benefits and facilitation of communication with students.

Foster started using Blackboard tools for a Public Health Studies class where she was grading student reflections. The subject matter—public health studies in the media—was outside of her field of physical chemistry. Blackboard facilitates creating a rubric that students can see when doing an assignment and the instructor then uses to grade that work. She showed the rubric detail that students see in Blackboard, and how the rubric can be used in grading. [See the CER Tutorial on Blackboard Rubrics and Rubrics-Helpful Hints] The rubric gives the students direction and assures that the instructor (or other graders) will apply the same standards across all student work.

It empowers students when they know exactly what criteria will be used in evaluating their work and how many points will be assigned to each component. Foster has found that using rubrics is an effective way to communicate assignment requirements to students, and that it helps her to clarify for herself what at the most important points. She noted that a rubric is very useful when there are multiple graders, such as Teaching Assistants (TAs), as it helps to calibrate the grading.

In response to questions from the audience, Foster stated that rubrics can be developed to cover both qualitative and quantitative elements. Developing good rubrics is an iterative process; it took her some time to sharpen her skills. There is flexibility in differentiating points allotted, but the instructor must be thoughtful, plan for a desired outcome, and communicate clearly. The rubric tool can be used to grade PDF files as well as Word documents. Foster noted that it is important to take opportunities to teach students to learn to write, learn to use technology, learn to read instructions, and learn to look at feedback given on assignments. Being transparent and explaining why you are using a particular technology will go a long way.

Reid Mumford gave his presentation on how he calibrates multiple graders (see slides). Mumford oversees the General Physics lab courses. This is a two semester, required sequence, so not all students are excited to be there. The sequences are on Mechanics and Electricity and Magnetism; both labs are taught every semester with multiple sections for each course. Approximately 600 to 700 students are taking these lab sequences each semester; students are divided into sections of about 24 students. The labs are open-ended and flexible, so students aren’t filling in blanks and checking boxes, which would be easier to grade. Lab sections are taught and graded by graduate student TAs, with about 30 TAs teaching each semester. Teaching and grading styles vary among these TAs as would be expected. Clearly, calibrating their grading is a challenge.

Grades are based on the best 9 of 10 lab activities, which consist of a pre-lab quiz and a lab note. All activities are graded using the same rubric. The grading scale used can be seen in the slides. One of the criteria for grading is “style,” which allows some flexibility and qualitative assessment. Students have access to the rubric, which is also shown in the slides.

About three years ago, Mumford adopted Turnitin (TII), the plagiarism detection tool, for Screen shot of Quick Mark grading tool.its efficient grading tools. It works well for his use because it is integrated with Blackboard. TII does its job in detecting cheating (and Mumford noted that lots of students are cheating), but it is the grading tools that are really important for the TAs. TAs are encouraged to be demanding in their grading and leave a lot of feedback, so grading takes them two to four hours each week. TII’s Feedback Studio (formerly known as GradeMark) allows TAs to accomplish their mission. [See CER tutorial on Feedback Studio and The Innovative Instructor post on GradeMark.] It was the QuickMark feature that sold Mumford on Feedback Studio and TII grading. Using the rubric for each activity, QuickMark can be pre-populated with commonly-used comments, which can then be dragged and dropped onto the student’s submitted work.

Graph showing General Physics Laboratory Section Grading Trends.These tools helped make the grading load more efficient, but calibrating the multiple graders was another challenge. Mumford found that the TAs need lots of feedback on their grading. Each week he downloads all the grades from Blackboard grade centers. He creates a plot that shows the average score for the weekly lab assignment. Outliers to the average scores are identified and these TAs are counseled so that their grading can be brought into line. Mumford also looks at section grading trends and can see which sections are being graded more leniently or harshly than average. He works with those TAs to standardize their grading.

In calculating final grades for the course, Mumford keeps three points in mind: final letter grades must be calculated, there should be no “easy” or “hard” sections of lab, and distribution should not vary (significantly) between sections. He makes use of per-section mapping and uses average and standard deviation to map results to a final letter grade model. Mumford noted that students are made aware, repeatedly, of the model being used. He is very transparent—everything is explained in the syllabus and reiterated weekly in lab sessions.

In conclusion, Mumford offered these take-aways:

  • Calibrating Multiple Graders is not easy
  • Tools are needed to handle multiple sections efficiently
  • Rubrics help but do not solve the calibration problem
  • Regular feedback to graders is essential
  • Limit of the system: student standing is ambiguous

In the future Mumford plans to give students a better understanding of course standing, to calculate a per-section curve each week, and to overcome some technical issues and the greater time investment that will be required with weekly calibrating and rescaling.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image Sources: Lunch and Learn Logo, slides from Mumford presentation

Considering the Use of Turnitin

Earlier this week an article from Inside Higher Ed (IHE) caught my eye. Sign with hand and text reading prevent plagiarism. In New Salvo Against Turnitin (June 19, 2017) Nick Roll summarizes an essay by Sean Michael Morris, Instructional Designer in the Office of Digital Learning at Middlebury College, and Jesse Stommel, Executive Director, Division of Teaching and Learning Technologies at the University of Mary Washington. The essay authors argue that faculty should rethink the use of Turnitin, questioning not only “…the control and use of people’s data by corporations…” but “…Turnitin’s entire business model, as well as the effects on academia brought on by its widespread popularity.” Morris and Stommel further contend that those using Turnitin “supplant good teaching with the use of inferior technology” reducing the student-instructor relationship to one where suspicion and mistrust are at the forefront. [Turnitin is a software application used to detect plagiarism, and Morris and Stommel are not the first to decry the company’s business model and practices.]

Although the IHE article provides a fair summary, as well as additional comments by Morris and Stommel, it is worth reading the 3,928 word essay—A Guide for Resisting Edtech: The Case Against Turnitin (Digital Pedagogy Lab, June 15, 2017)—to appreciate the complex argument. I agree with some of the concerns the authors address and feel we should be doing more individually and collectively to school ourselves and our students in the critical evaluation of digital tools, but disagree with what I feel are over-simplifications and unfair assumptions. Morris and Stommel cast faculty who use Turnitin as “surrendering efficiency over complication” by not taking the time and effort to use plagiarism as a teachable moment. Further, they state that Turnitin takes advantage of faculty who are characterized as being, at the core, mistrustful of students.

The assumption that faculty using Turnitin are not actively engaging in conversations around and instruction of ethical behavior, including plagiarism, and are not using other tools and resources in these activities is simply not correct. The assertion that faculty using Turnitin are suspicious teachers who are embracing an easy out via an efficient educational technology is also not accurate.

The reality is that some students will plagiarize, intentionally or not, and the Internet, social media practices, and cultural differences have rendered complicated students’ understanding of intellectual property. I believe that many of our institutions of higher learning, and faculty and library staff therein, make concerted efforts to teach students about academic integrity. This includes the meaning and value of intellectual property, as well as finer points of what constitutes plagiarism and strategies to avoid it.

I believe it is relevant to note that Middlebury College’s website boasts a mean class size of 16, while the University of Mary Washington lists an average class size of 19. Student-faculty ratios are 8 to1 and 14 to 1 respectively.  I cannot help but feel that Morris and Stommel are speaking from a point of privilege working in these two institutions. Instructors who teach at large, underfunded, state universities with classes of hundreds of students, relying on a corps of teaching assistants to grade their essays, are in a different boat.

The authors state: “So, if you’re not worried about paying Turnitin to traffic your students’ intellectual property, and you’re not worried about how the company has glossed a complicated pedagogical issue to offer a simple solution, you might worry about how Turnitin reinforces the divide between teachers and students, short-circuiting the human tools we have to cross that divide.” In fact, we may all be worried about Turnitin’s business model and be seeking a better solution. Yet in this essay nothing more concrete is given us on those human tools and how faculty in less privileged circumstances can realistically and effectively make use of them.

The Innovative Instructor has in the past posted on Teaching Your Students to Avoid Plagiarism (November 5, 2012, Macie Hall), and using Turnitin as a teaching tool: Plagiarism Detection: Moving from “Gotcha” to Teachable Moment (October 9, 2013, Brian Cole and Macie Hall). These articles may be helpful for faculty struggling with the issues at hand.

Yes, we should all be critical thinkers about the pedagogical tools we use; in the real world, sometimes we face hard choices and must fall back on less than ideal solutions.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image source: Microsoft Clip Art edited by Macie Hall

Quick Tips: Tools for Creating Rubrics

The Innovative Instructor has previously shared posts on the value of using rubrics (Creating Rubrics, Louise Pasternak, November 21, 2014 and Sharing Assignment Rubrics with Your Students, Macie Hall June 26, 2014). Today’s Quick Tips post offers some tools and resources for creating rubrics.

Red sharpie-type marker reading "Rubrics Guiding Graders: Good Point" with an A+ marked below

Red Rubric Marker

If you are an instructor at Johns Hopkins or another institution that uses the Blackboard learning management system or Turnitin plagiarism detection, check out these platforms for their built-in rubric creation applications. Blackboard has an online tutorial here. Turnitin offers a user guide here.

If neither of these options are available to you, there is a free, online application called Rubistar that offers templates for rubric design based on various disciplines, projects, and assignments. If none of the templates fit your need, you can create a rubric from scratch. You must register to use Rubistar. A tutorial is available to get you started. And you can save a printable rubric at the end of the process.

Wondering how others in your field have designed rubrics for specific assignments or projects? Google for a model: e.g., “history paper rubric college,” “science poster rubric college,” “video project rubric college” will yield examples to get your started. Adding the word “college” to the search will ensure that you are seeing rubrics geared to an appropriate level.

With free, easy to use tools and plentiful examples to work from, there is no excuse for not using rubrics for your course assignments.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image source © 2014 Reid Sczerba

 

 

Quick Tips: Paperless Grading

Just in time for the end of semester assignment and exam grading marathon, The Innovative Instructor has some tips for making these tasks a bit less stressful.

Male instructor 's head between two stacks of papers.Last year we wrote about the GradeMark paperless grading system, a tool offered within Turnitin, the plagiarism detection software product used at JHU. The application is fully integrated with Blackboard, our learning management system. For assignments and assessments where you don’t wish to use Turnitin, Blackboard offers another grading option for online submissions. Recent updates to Blackboard’s include new features built into the assignment tool that allow instructors to easily make inline comments, highlight or strikeout text, and use drawing tools for freeform edits. All this without having to handle a single piece of paper.

If you don’t use Blackboard, don’t despair. The Innovative Instructor has solutions for you, too.  A recent post in one of our favorite blogs, the Chronicle of Higher Education’s Professor Hacker, titled Using iAnnotate as a Grading Tool, offers another resource. According to its creators, the iAnnotate app “turns your tablet into a world-class productivity tool for reading, marking up, and sharing PDFs, Word documents, PowerPoint files, and images.” This means that if you students submit documents in any of these formats (Professor Hacker suggests using DropBox, Sky Drive, Google Drive, or other cloud storage services for submission and return of assignments), you can grade them on your iPad using iAnnotate.

Erin E. Templeton, Anne Morrison Chapman Distinguished Professor of International Study and an associate professor of English at Converse College and author of the post, has this to say about how she uses iAnnotate’s features.

With iAnnotate, you can underline or highlight parts of the paper. I will often highlight typos, sentences that are unclear, or phrases that I find especially interesting. I can add comments to the highlight to explain why I’ve highlighted that particular word or phrase. You can also add comment boxes to make more general observations or ask questions, or if you would prefer, you can type directly on the document and adjust the font, size, and color to fit the available space.

I frequently use the stamp feature, which offers letters and numbers (I use these to indicate scores or letter grades), check marks, question marks, stars of various colors, smiley faces–even a skull and crossbones…. And if you’d rather, you can transform a word or phrase that you find yourself repeatedly tying onto the document into a stamp–I have added things like “yes and?” and “example?” to my collection. Finally, there is a pencil tool for those who want to write with either a stylus or a finger on the document.

Not an iDevice user? iAnnotate is available for Androids too, although it is limited at the time of this posting to reading and annotating PDF files.

The Professor Hacker post offers additional links and resources for paperless grading and more generally for those looking to move to a paperless course environment.  Be sure to read the comments for additional solutions.

Macie Hall, Senior Instructional Designer
Center for Educational Resources


Image Source: Microsoft Clip Art

Plagiarism Detection: Moving from “Gotcha” to Teachable Moment

Parts of this post appeared in our The Innovative Instructor print series with the title Turnitin by CER staff member Brian Cole.

A previous The Innovative Instructor post on preventing plagiarism gave links to websites with guides, tutorials, and activities.

Sign with hand and text reading prevent plagiarism.Integrity is a core value for every academic community. Here at Johns Hopkins training students on ethical behavior including plagiarism begins at freshman orientation. However, the importance of proper citation and use of paraphrasing and quotations are not learned in a single session. While our librarians offer ongoing support, both directly to students and by working with faculty in the classroom providing modules on research resources and specific citation standards, improper citation practices and outright plagiarism continue to be a problem at our campus and elsewhere.

Part of the problem is the ease of cutting and pasting that comes along with unparalleled access to online content. Although resources on avoiding plagiarism are available to students, often they do not have a good understanding of proper quotation and paraphrasing techniques or when and how to cite borrowed material. On the other side, it is cumbersome for instructors to check submitted papers for originality against online sources. At a certain point, particularly in courses with large enrollments, the process of checking suspect papers using a Google search becomes unmanageable, and some content will not show up using standard search engines.

Enter plagiarism detection software applications. These applications have gained popularity in the higher education community as easily available online source material has proliferated. Googling for “plagiarism checker” will yield links to a number of applications, including some that are free. At Johns Hopkins, we have a license for the widely-used application known as Turnitin. Turnitin is a web-based service for detecting plagiarism and improper citations in student-submitted work.

Some faculty have been reluctant to turn to a plagiarism detection tool feeling that it creates an atmosphere of distrust in the classroom. But rather than seeing it as a “gotcha” faculty should know that Turnitin’s value goes beyond simply identifying plagiarism in student papers. The reports produced allow instructors to flag misunderstandings as to proper usage of borrowed content and direct students to remedial resources. Turnitin can be an excellent teaching tool.

Turnitin’s Originality Report does not judge whether a student has plagiarized. Rather, it shows what percentage of a paper’s text matches a source and what source it matches. It is then up to the instructor to decide whether the matches are acceptable, whether they are the result of improper citations, or if they constitute inappropriate use of others’ works.

Instructors can decide on several variables for each assignment, such as whether students can see the Originality Report and resubmit papers. Writing classes often use these options to teach proper citation.

It’s worth noting that in the past there have been controversies surrounding the use of Turnitin and similar services. Students have contended that it is illegal for these companies to keep their papers in its database and accused them of improperly deriving profit from student submitted work. Turnitin has weathered these controversies and prevailed in court challenges, mainly because they do not publish the student submissions but only use them for matching.

Knowing that their papers will be checked sends the message to your students that they need to be mindful of proper citation practices. As a best practice, it is recommended that you not single out individual papers for checking as then all students are not subject to the same scrutiny. Rather, all student papers from a given assignment should be submitted for plagiarism detection.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Brian Cole, Senior Information Technologist
Center for Educational Resources


Image Source: Microsoft Clip Art edited by Macie Hall

GradeMark Paperless Grading

GradeMark is a paperless grading system that gives instructors the ability to add comments and corrections to assignments submitted electronically. It is a tool offered within Turnitin, the plagiarism detection software product used at JHU. With its drag and drop functionality, among other features, GradeMark has the potential to save instructors a great deal of time when grading online assignments.  It is also easily integrated with Blackboard.

(Note: In order to use GradeMark, online assignments must be created using Turnitin. If using Turnitin within Blackboard, accounts are automatically created for instructors and students through the Blackboard system. If using Turnitin outside of Blackboard, the instructor is responsible for creating separate accounts for each student. Please click here for more information on Turnitin’s integration with Blackboard.)

Screen shot showing example of using GradeMark

GradeMark contains several different grading features:

  • Dragging and Dropping Quickmarks – Quickmarks are frequently used comments that are readily available to drag and drop into a student’s assignment. While viewing an assignment, the instructor can select from a panel of standard Quickmarks that come with GradeMark, or from a custom set that s/he has created.  For example, the abbreviation ‘Awk.’ is a Quickmark indicating an awkward phrase. The ability to drag and drop Quickmarks to an assignment, instead of typing them over and over again, can save instructors a lot of time.
  • General Comments – Each assignment has a generous space where general comments can be added.  General comments can be used to further clarify any Quickmarks that were added as well as discuss the assignment as a whole.
  • Voice Comments – A recent addition to GradeMark is the ability to add voice comments. A voice comment can be added to the assignment lasting up to three minutes in length.  An instructor can use the built-in microphone in his/her computer to easily record the message.
  • Rubrics – Rubrics created within GradeMark can help streamline the grading process by using a ‘scorecard’ approach. Specific criteria and scores are defined in a rubric that is then associated with an assignment. Instructors grade the assignment by filling in the scores based on the evaluative criteria in the rubric. There is also the option of associating Quickmarks with rubrics when they are added to the assignment.

Students are able to view their graded assignments when the ‘post date’ is reached. The post date is set by the instructor when setting up the assignment. Students have the option to print or save a copy of the graded assignment and can view only their own submissions.

GradeMark Logo showing grade book and apple

Advantages:

  • Flexibility in marking up assignments – Quickmarks, rubrics, text, voice comments all available.
  • Time saved dragging and dropping reusable comments.
  • Increased consistency in grading.
  • Clear feedback to students, instead of ‘scribbled margins.’
  • Opportunity to provide more detailed feedback to students including links and resources.
  • No need to download assignments – everything is web-based, stored online.
  • If the instructor is using Blackboard, when the assignment is graded the grade is automatically transferred and recorded into the Blackboard Grade Center.

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image sources: Amy Brusini screen shot of GradeMark example; GradeMark logo

Teaching Your Students to Avoid Plagiarism

As the semester passes the midterm mark and papers and reports come due, we begin to get requests from faculty for ways to teach students how to avoid plagiarism. Most often students plagiarize unintentionally, because they don’t know how to cite sources properly, cut and paste from e-resources, and aren’t skilled in the arts of paraphrase and summary.

Recently a colleague, Lynne Stuart, the MSEL Librarian for Economics, Government, Law, Policy Studies, pointed me to a great web site on plagiarism at Arizona University that addresses this, and is called, appropriately, Accidental Plagiarism. There are two tutorials that provide background information on what plagiarism is and provide examples of how to properly summarize, paraphrase, and quote sources. The first has a sidebar menu for navigating; the second is an interactive tutorial that resembles a series of slides. In both cases students can practice skills and test themselves.

Sign with hand and text reading prevent plagiarism.Google search “plagiarism exercises college” yielded many, many more examples. Here’s an editor’s pick of some of the best.

Harvard University’s Graduate School of Education offers the Principles of Paraphrasing, How to Avoid Inadvertent Plagiarism in Three Easy Modules, which is pretty slick and comprehensive. The format is PowerPoint with audio, worksheets with answer keys and handouts. Exercises are included.

Cornell University’s College of Arts and Sciences recognizing and avoiding plagiarism site is divided into three sections: principles, logistics (how to recognize and check for plagiarism), and exercises. It provides a good overview, plus the exercises (quizzes), which you can take as a guest. The quizzes cover a variety of disciplinary examples.

Indiana University School of Education Understanding Plagiarism site provides an overview, links to real plagiarism cases, plagiarism examples and explanations, self-practice, and a test that is available for non-IU visitors.

The University of Southern Mississippi, University Libraries’ Plagiarism Tutorial has a tutorial adapted from  tutorial was adapted from Robert A. Harris’s The Plagiarism Handbook : Strategies for Preventing, Detecting, and Dealing with Plagiarism (Los Angeles, CA : Pyrczak Publishing, 2001), combined with a true-false pre-test and two interactive quizzes.

WISC-Online [Wisc-Online is a digital library of Web-based learning resources called “learning objects.”] There is a short learning object on plagiarism that provides a basic overview then presents six examples for self-testing.

Purdue University’s OWL [online Writing Lab] is a great general resource for scholarly writing. It includes sections on using research you’ve conducted in your writing: Quoting, Paraphrasing and Summarizing, Paraphrasing, a Paraphrasing Sample Essay, Paraphrase Exercises, and Avoiding Plagiarism. There are some plagiarism exercises; however these are less useful than those found on other sites, as there are no answers or comments provided. These are meant for class discussion.

The Center for Educational Resources also supports Turnitin, a plagiarism prevention application. Find out more about using Turnitin at JHU on the CER website.

And don’t forget the Sheridan Libraries Research Help where you can find information on a variety of topics including proper citation and evaluating materials found on the Internet.

Macie Hall, Senior Instructional Designer
Center for Educational Resources


Image Source: Microsoft Clip Art edited by Macie Hall