Lunch and Learn: Creating Rubrics and Calibrating Multiple Graders

Posted on December 28, 2017 by Macie Hall

On Friday, December 15, the Center for Educational Resources (CER) hosted the second Lunch and Learn—Faculty Conversations on Teaching—for the 2017-2018 academic year. Laura Foster, Academic Advisor, Public Health Studies, and Reid Mumford, Instructional Resource Advisor, Physics & Astronomy, presented on Creating Rubrics and Calibrating Multiple Graders.

Laura Foster led by giving us a demonstration of her use of Blackboard for creating rubrics. She noted that she might be “preaching to the choir” but hoped that those present might take back these best practices to their colleagues. Noting that many faculty have negative opinions of Blackboard, she put in a plug for its organizational benefits and facilitation of communication with students.

Foster started using Blackboard tools for a Public Health Studies class where she was grading student reflections. The subject matter—public health studies in the media—was outside of her field of physical chemistry. Blackboard facilitates creating a rubric that students can see when doing an assignment and the instructor then uses to grade that work. She showed the rubric detail that students see in Blackboard, and how the rubric can be used in grading. [See the CER Tutorial on Blackboard Rubrics and Rubrics-Helpful Hints] The rubric gives the students direction and assures that the instructor (or other graders) will apply the same standards across all student work.

It empowers students when they know exactly what criteria will be used in evaluating their work and how many points will be assigned to each component. Foster has found that using rubrics is an effective way to communicate assignment requirements to students, and that it helps her to clarify for herself what at the most important points. She noted that a rubric is very useful when there are multiple graders, such as Teaching Assistants (TAs), as it helps to calibrate the grading.

In response to questions from the audience, Foster stated that rubrics can be developed to cover both qualitative and quantitative elements. Developing good rubrics is an iterative process; it took her some time to sharpen her skills. There is flexibility in differentiating points allotted, but the instructor must be thoughtful, plan for a desired outcome, and communicate clearly. The rubric tool can be used to grade PDF files as well as Word documents. Foster noted that it is important to take opportunities to teach students to learn to write, learn to use technology, learn to read instructions, and learn to look at feedback given on assignments. Being transparent and explaining why you are using a particular technology will go a long way.

Reid Mumford gave his presentation on how he calibrates multiple graders (see slides). Mumford oversees the General Physics lab courses. This is a two semester, required sequence, so not all students are excited to be there. The sequences are on Mechanics and Electricity and Magnetism; both labs are taught every semester with multiple sections for each course. Approximately 600 to 700 students are taking these lab sequences each semester; students are divided into sections of about 24 students. The labs are open-ended and flexible, so students aren’t filling in blanks and checking boxes, which would be easier to grade. Lab sections are taught and graded by graduate student TAs, with about 30 TAs teaching each semester. Teaching and grading styles vary among these TAs as would be expected. Clearly, calibrating their grading is a challenge.

Grades are based on the best 9 of 10 lab activities, which consist of a pre-lab quiz and a lab note. All activities are graded using the same rubric. The grading scale used can be seen in the slides. One of the criteria for grading is “style,” which allows some flexibility and qualitative assessment. Students have access to the rubric, which is also shown in the slides.

About three years ago, Mumford adopted Turnitin (TII), the plagiarism detection tool, for its efficient grading tools. It works well for his use because it is integrated with Blackboard. TII does its job in detecting cheating (and Mumford noted that lots of students are cheating), but it is the grading tools that are really important for the TAs. TAs are encouraged to be demanding in their grading and leave a lot of feedback, so grading takes them two to four hours each week. TII’s Feedback Studio (formerly known as GradeMark) allows TAs to accomplish their mission. [See CER tutorial on Feedback Studio and The Innovative Instructor post on GradeMark.] It was the QuickMark feature that sold Mumford on Feedback Studio and TII grading. Using the rubric for each activity, QuickMark can be pre-populated with commonly-used comments, which can then be dragged and dropped onto the student’s submitted work.

These tools helped make the grading load more efficient, but calibrating the multiple graders was another challenge. Mumford found that the TAs need lots of feedback on their grading. Each week he downloads all the grades from Blackboard grade centers. He creates a plot that shows the average score for the weekly lab assignment. Outliers to the average scores are identified and these TAs are counseled so that their grading can be brought into line. Mumford also looks at section grading trends and can see which sections are being graded more leniently or harshly than average. He works with those TAs to standardize their grading.

In calculating final grades for the course, Mumford keeps three points in mind: final letter grades must be calculated, there should be no “easy” or “hard” sections of lab, and distribution should not vary (significantly) between sections. He makes use of per-section mapping and uses average and standard deviation to map results to a final letter grade model. Mumford noted that students are made aware, repeatedly, of the model being used. He is very transparent—everything is explained in the syllabus and reiterated weekly in lab sessions.

In conclusion, Mumford offered these take-aways:

Calibrating Multiple Graders is not easy
Tools are needed to handle multiple sections efficiently
Rubrics help but do not solve the calibration problem
Regular feedback to graders is essential
Limit of the system: student standing is ambiguous

In the future Mumford plans to give students a better understanding of course standing, to calculate a per-section curve each week, and to overcome some technical issues and the greater time investment that will be required with weekly calibrating and rescaling.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image Sources: Lunch and Learn Logo, slides from Mumford presentation

Thinking about Accessibility Part 2

Posted on December 15, 2017 by Macie Hall

Last week ‘s post summarized the first part of an important article by Anne-Marie Womack, Teaching Is Accommodation: Universally Designing Composition Classrooms and Syllabi, College Composition and Communication, February 2017. Womack, assistant director of writing at Tulane University, encourages instructors to rethink accommodation as normative rather than an exception to a rule.

The second part of the essay focuses on creating an accessible syllabus both in theory and in practice by “moving from syllabus as contract to syllabus as accommodation.” This can be accomplished in a number of ways, starting by not limiting the notion of accommodation to a paragraph stating the institutional disability policy. Syllabi are often heavy on text and present inflexible policies written in a punitive and/or defensive tone. Womack suggests instead that instructors strive to create accessible documents that engage students with “cooperative language” and flexibility in assignments. She does acknowledge that it is important to consider the particular student audience in designing a syllabus—one solution doesn’t fit all. Instead, she suggests three strategies.

The first strategy is Creating Accessible Document Design. Reducing the amount of text in your syllabus is the primary consideration. Womack starts with the boilerplate policies that may be required by the institution, school, or department. Such information may be reduced to basics, linked to, or included at the end in an appendix. Other information can be provided during the semester as it is needed, e.g., assignment specifics, writing prompts.

How the text is formatted can make a big difference—Womack suggests trading some text for accessible images. This will necessitate the creation of alternative text (alt text) to describe the image for those who use screen readers. She suggests that alt text be “focused on data, not extraneous visual details.” Since images can introduce color that may not be visible to all readers, there are guidelines for best practices. (I’ve included some resources for creating accessible images and documents at the end of this post.) Womack shows examples of how she converted her syllabus from text heavy to accessible by using images, blocks of text, bulleted lists, and icons, and increasing the amount of white space. Using a larger font size and a san serif font will also improve readability. Your goal is to “make the syllabus user-friendly” because you want your students to use it. Other tips include creating a table of contents for improved navigation, making internal hyperlinks to connect items within the syllabus, using headings to establish hierarchy, bolding text to emphasize key ideas.

The second strategy Womack advocates is Engaging Students with Cooperative Language. You want to convey “approachability and empathy” rather than making students feel uncomfortable requesting accommodations. Don’t focus on negative consequences and punitive rules. Instead look at inclusive practices in your syllabus as a means to engage student cooperation.

Womack suggests beginning with an “inclusive learning statement” and offers an example:

Your success in this class is important to me. We will all need accommodations because we all learn differently. If there are aspects of this course that prevent you from learning or that form barriers to your inclusion, please let me know as soon as possible. Together we’ll develop strategies that can enable you to succeed in the course.

I encourage you to visit the Office of Disability Services to determine how you could improve your learning as well. If you need official accommodations, you have a right to have these met. There is also a range of resources on campus, including the Writing Center, Tutoring Center, and Academic Advising Center. (Figure 5, p.513).

Further, she gives examples of positive versus punitive language, commands versus invitations, and cooperative versus paternalistic language.

The third strategy is labeled Empowering Students through Flexible Course Plans. Womack saw that “traditional accommodations, such as longer time and adaptable assessment, provide a starting point to improve course practices.” Citing a number of research studies, she states that students are more motivated to learn when they feel they have autonomy. Womack suggests ways in which instructors can give students control. She discusses allowing students flexibility in deadlines. Her approach combines student-set and instructor-set deadlines, which allows an extended time-frame but prevents students from falling behind. Low-stakes writing assignments and incremental assignments may also provide flexibility. Minimal grading for some of this work is recommended. Womack also examines building flexibility into grading distributions. She focuses on those which work well for her writing courses, including contract grading where a system of assessment is negotiated. While this aspect of flexibility may be more difficult to incorporate into large lecture courses, the point is that assessments and grading should be carefully considered. Allowing students to drop the lowest grade in a series of quizzes or homeworks is an easy means of allowing students flexibility and control in a larger class.

In concluding Womack reminds us that accommodation does not give disabled students an unfair advantage; “it is more likely all students have been given an unfair disadvantage through inaccessible pedagogy.” We should look at the “best versions of accommodation” as a way of teaching inclusively to a diverse audience and adapting in response to individual needs.

While Womack offers some good examples in her essay, there are some great resources with “how-to” guides available. Womack created a website at Tulane University on making an accessible syllabus that frequently cited, but alas, no longer “accessible.” However, you can view an archived image of the site here.

If you are someone who likes video tutorials, the University of Minnesota Accessible U has a series of video tutorials on creating an accessible syllabus and making accessible documents and accessible PDFs.

You will also want to check out the University of Colorado Boulder Universally Designed Syllabus Materials website “to employ best practices when creating a course syllabus.” Some of the material is UC system-specific, but much of the information is widely applicable.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image Source: Pixabay.com

Thinking about Accessibility Part 1

Posted on December 6, 2017 by Macie Hall

Often I read an article or blog post and suddenly find that I am falling down a rabbit hole. Hours, or even days, later I emerge, having uncovered a wealth of information and resources that have to be edited down in order to present a reasonably digestible overview of a topic for you readers. Such was the case with David Gooblar’s post Now is the Time to Think About Accessibility on his Chronicle Vitae Pedagogy Unbound blog (August 8, 2017). From the links in his article, I went on to discover other great material on accessibility to share—enough good stuff for two posts. This first post will cover thinking about accessibility in your classroom in general; next will be a follow-up post focused on creating an accessible syllabus and other documents for your class.

Gooblar starts off by noting that for many instructors, accessibility is given a brief mention at the end of the syllabus and then forgotten. Accessibility is seen as “an exception to the norm” and given little thought. He then notes an article by Anne-Marie Womack, assistant director of writing at Tulane University, which takes issue with that way of “conceptualizing accessibility.”

Teaching Is Accommodation: Universally Designing Composition Classrooms and Syllabi (College Composition and Communication, February 2017) by Anne-Marie Womack should be required reading for all higher ed instructors. [Note: if the link does not work, try copying and pasting this URL directly into your browser: http://www.ncte.org/library/NCTEFiles/Resources/Journals/CCC/0683-feb2017/CCCC0683Teaching.pdf.] It is an important document that asks us to rethink disability and academic accommodations. She starts by discussing “contemporary theories of disability to retheorize accommodation as the process of teaching itself.” Womack provides a history of disability law and American institutions of higher education, noting that students today must “pass substantial hurdles to qualify for accommodations” often at the risk of being stigmatized. Faculty, who may receive little institutional support, come to feel that they are the ones burdened by the process. Any resulting pedagogical changes are seen as affecting only the students with disabilities.

Womack argues that resistance to accommodation by university administration and faculty assumes that accommodations are an exception to a rule, to a best practice, or normal way of teaching. Womack states, “Ultimately, though, there is no normal, primary way of learning, only normalized methods made primary through frequent use. Material always changes as it moves from expert to novice. Every act of teaching is an accommodation because it creates certain conditions for students to learn and display learning.” Even though effective student learning means that the material is accessible, instructors have come to feel that “making material accessible to disabled students threatens academic rigor.”

Seen from another vantage point, inclusive teaching means eliminating barriers to learning, not eliminating intellectual challenges. Womack says, “Accommodation is the most basic act and art of teaching. It is not the exception we sometimes make in spite of learning, but rather the adaptations we continually make to promote learning.” She advocates accommodation of disabled students within a universal design framework.

“Universal Design is the design and composition of an environment so that it can be accessed, understood and used to the greatest extent possible by all people regardless of their age, size, ability or disability.” Universal design is good for everyone. In your classroom, Womack suggests, for example, considering guidelines for dyslexic and blind readers, working under the assumption that by creating documents that more students can read, more students will read. She warns, however, that universal design must be used as a process, not to negate the need for accommodations, but to start to negotiate the means to accessibility for all.

In my next post I will look at the second part of Womack’s article, which provides suggestions for creating accessible documents, engaging students by using “cooperative language” and building flexibility into your course to empower students. “If instructors see the syllabus through the lens of disability, then the question becomes not how policies protect a normative standard but how far they extend inclusion.”

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image Source: Pixabay.com

The Innovative Instructor

Pedagogy – Best Practices – Technology

Monthly Archives: December 2017

Lunch and Learn: Creating Rubrics and Calibrating Multiple Graders

Thinking about Accessibility Part 2

Thinking about Accessibility Part 1