Monthly Archives: November 2014

Managing Teamwork with CATME

Posted on by
1

Many instructors recognize the value of having students work collaboratively on team-based assignments. Not only is it possible for students to experience a greater understanding of the subject material, but several life-long learning skills can be gained through active engagement with team members. Managing team-based assignments, however, is not something most instructors look forward to; the administrative tasks can be quite cumbersome, especially with large classes. Thankfully there is a tool to help with this process: CATME.

Logo for CATMECATME, which stands for ‘Comprehensive Assessment of Team Member Effectiveness,’ is a free set of tools designed to help instructors manage group work and team assignments more effectively. It was developed by a diverse group of professors with extensive teaching experience, as well as researchers and students. First released in 2005, CATME takes away much of the administrative burden that instructors face when trying to organize and manage teams, communicate with students, and facilitate effective peer evaluation.

‘Team Maker,’ one of two main parts of CATME, assists with the team creation process. First, it allows instructors to easily create and send a survey to students. The survey collects various demographic data, previously completed coursework, and student availability information. Instructors can also add their own questions to the survey if desired. Once the data are collected, instructors decide which criteria will be used to create the teams and then assign weights to each of the criterion. Team Maker then uses the weights in an algorithm to create the teams.  Instructors are free to adjust the teams, if necessary, to their satisfaction. Once the teams are finalized, the instructor releases the results to students, who are provided with their team members’ names, email addresses, and a schedule matrix showing member availability.

‘Peer Evaluation,’ the other core component of CATME, is used by students to evaluate their teammates’ performance as well as their own.  The web-based ratings page is presented on one screen, making it easy to fill out and submit results. Students select from a set of behaviors which most closely describes themselves and their peers. There is also a place where students can include confidential comments which are only seen by the instructor.  Once completed, instructors can decide when to release the evaluation results to students. Peer ratings appear anonymous to students but are identified for instructors.

Another tool included in CATEME is the ‘Rater Calibration’ tool, which helps train students in the peer evaluation process. Students are asked to rate a series of fictional team members and then receive feedback about their ratings. Other tools include the ‘Student Team Training’ tool, designed to help students recognize effective team behaviors, and the ‘Meeting Support’ tool, which provides templates that students can use to plan and organize meetings, such as writing a team charter, taking minutes, etc.

To view a video demo of CATME and learn more about the product, visit the CATME website. Instructors interested in using CATME can go to https://www.catme.org/login/request to register for an account.

Amy Brusini, Course Management Training Specialist Center for Educational Resources

Image Source: CATME logo from http://info.catme.org/

Creating Rubrics

Posted on by
2
Red sharpie-type marker reading "Rubrics Guiding Graders: Good Point" with an A+ marked below

Red Rubric Marker

Instructors have many tasks to perform during the semester. Among those is grading, which can be subjective and unstructured. Time spent constructing grading rubrics while developing assignments benefits all parties involved with the course: students, teaching assistants and instructors alike. Sometimes referred to as a grading schema or matrix, a rubric is a tool for assessing student knowledge and providing constructive feedback. Rubrics are comprised of a list of skills or qualities students must demonstrate in completing an assignment, each with a rating criterion for evaluating the student’s performance. Rubrics bring clarity and consistency to the grading process and make grading more efficient.

Rubrics can be established for a variety of assignments such as essays, papers, lab observations, science posters, presentations, etc. Regardless of the discipline, every assignment contains elements that address an important skill or quality. The rubric helps bring focus to those elements and serves as a guide for consistent grading that can be used from year to year.

Whether used in a large survey course or a small upper-level seminar, rubrics benefit both students and instructors. The most obvious benefit is the production of a structured, consistent guideline for assigning grades. With clearly established criteria, there is less concern about subjective evaluation. Once created, a rubric can be used every time to normalize grading across sections or semesters. When the rubric for an assignment is shared with teaching assistants, it provides guidance on how to translate the instructor’s expectations for evaluating student submissions consistently. The rubric makes it easier for teaching assistants to give constructive feedback to students. In addition, the instructor can supply pre-constructed comments for uniformity in grading.

Some instructors supply copies of the grading rubric to their students so they can use it as a guide for completing their assignments. This can also reduce grade disputes. When discussing grades with students, a rubric acts as a reminder of important aspects of the assignment and how each are evaluated.

Below are basic elements of rubrics, with two types to consider.

I. Anatomy of a rubric

All rubrics have three elements: the objective, its criteria, and the evaluation scores.

Learning Objective
Before creating a rubric, it is important to determine learning objectives for the assignment. What you expect your students to learn will be the foundation for the criteria you establish for assessing their performance. As you are considering the criteria or writing the assignment, you may revise the learning objectives or adjust the significance of the objective within the assignment. This iteration can help you hone in on what is the most important aspect of the assignment, choose the appropriate criteria, and determine how to weigh the scoring.

Criteria
When writing the criteria (i.e., evaluation descriptors), start by describing the highest exemplary result for the objective, the lowest that is still acceptable for credit, and what would be considered unacceptable. You can express variations between the highest and the lowest if desired. Be concise by using explicit verbs that relate directly to the quality or skill that demonstrates student competency. There are lists of verbs associated with cognitive categories found in Bloom’s taxonomy (Knowledge, Comprehension, Application, Evaluation, Analysis, and Synthesis). These lists express the qualities and skills required to achieve knowledge, comprehension or critical thinking (Google “verbs for Bloom’s Taxonomy”).

Evaluation Score
The evaluation score for the criterion can use any schema as long as it is clear how it equates to a total grade. Keep in mind that the scores for objectives can be weighted differently so that you can emphasize the skills and qualities that have the most significance to the learning objectives.

II. Types of rubrics

There are two main types of rubrics: holistic (simplistic) and analytical (detailed).

Selecting your rubric type depends on how multi-faceted the tasks are and whether or not the skill requires a high degree of proficiency on the part of the student.

Holistic rubric
A holistic rubric contains broad objectives and lists evaluation scores, each with an overall criterion summary that encompasses multiple skills or qualities of the objective. This approach is more simplistic and relies on generalizations when writing the criteria.

The criterion descriptions can list the skills or qualities as separate bullets to make it easier for a grader to see what makes up an evaluation score. Below is an example of a holistic rubric for a simple writing assignment.

Table showing an example of a holistic rubric

Analytical rubric
An analytical rubric provides a list of detailed learning objectives, each with its own rating scheme that corresponds to a specific skill or quality to be evaluated using the criterion. Analytical rubrics provide scoring for individual aspects of a learning objective, but they usually require more time to create. When using analytical rubrics, it may be necessary to consider weighing the score using a different scoring scale or score multipliers for the learning objectives. Below is an example of an analytical rubric for a chemistry lab that uses multipliers.

Table showing an example of an analytical rubric

It is beneficial to view rubrics for similar courses to get an idea how others evaluate their course work. A keyword search for “grading rubrics” in a web search engine like Google will return many useful examples. Both Blackboard and Turnitin have tools for creating grading rubrics for a variety of course assignments.

Louise Pasternack
Teaching Professor, Chemistry, JHU

Louise Pasternack earned a Ph.D. in chemistry from Johns Hopkins. Prior to returning to JHU as a senior lecturer, Louise Pasternack was a research scientist at the Naval Research Laboratory. She has been teaching introductory chemistry laboratory at JHU since 2001 and has taught more than 7000 students with the help of more than 250 teaching assistants. She became a teaching professor at Hopkins in 2013.

Image sources: © 2014 Reid Sczerba

Writing to Learn

Posted on by
1

I’ve been touting the CIRTL (Center for the Integration of Research, Teaching, and Learning) MOOC, An Introduction to Evidence-Based Undergraduate STEM Teaching, for several weeks now. The course is coming to an end, but I am mining the materials for content to summarize here at The Innovative Instructor in case you missed it.

Students doing group workLast week the unit on Writing to Learn was particularly compelling. Janet L. Littrell, Ed.D, the Director of Distance Learning and Associate Director of the Engineering Education Research Center at the Swanson School of Engineering, University of Pittsburgh, taught the module. The material presented below is taken from the three videos Littrell produced.

The concept of writing to learn has been around since the 1970s, but has gained traction again more recently. The concept is to view writing as part of the learning process, not solely for the purpose of communicating information, but also as a reflective practice to increase student understanding, enhance learning, and provide instructors with feedback.

How does writing to learn differ from other writing students are asked to do as part of their coursework? Traditional writing assignments usually are done outside of class, are complete when turned in, are graded and returned to the students, and have the purpose of documenting students’ knowledge and comprehension.

Writing to learn assignments are often assigned and completed in class, are short, open-ended, may or may not be turned in, typically are not graded, and have the purpose of helping students think for themselves. Engagement is the goal, errors are ok. The idea is that students are encouraged to explore, question, develop their ideas, and/or reflect on their experiences. A writing to learn assignment is often a jumping-off point; it marks a beginning of a thought process rather than an end product. This type of writing is often referred to as low-stakes writing.

The goal of low stakes writing is to turn students into active learners, to help them find their own voices, and to focus on thoughts and ideas rather than on a formal writing structure. Have your students do smaller, more frequent writing assignments that are not graded. For example, have students keep a journal or learning log to document their ideas, thoughts, reactions, and to comment on class discussions, labs, readings and other assignments. At the beginning of class give students 5 minutes to free-write on a specified topic as a way of helping them gather their thoughts for a discussion. Take a minute or two at the end of class for students to write questions or comments they have on the day’s lecture or discussion. Or, if you sense that students may not be understanding what you are teaching, you can ask for mid-lecture feedback. Although writing to learn assignments are not usually graded, in these last two cases, where the responses provide formative assessment, the instructor should collect and read through them. In other cases, there might be a check plus/check minus system for completion of a writing assignment, with points that accumulate for credit over the course of the semester. You might also consider peer review for a writing to learn assignment.

Using low stakes writing or writing to learn assignments in your classes does not preclude having students write in more traditional ways. You should consider your learning objectives and assign writing accordingly. Consider, however, that the more students write, the better writers they will become. Low stakes writing helps them to understand that putting their thoughts on paper is part of a larger scholarly process involving inquiry, analysis, and critical thinking.

For more on writing to learn see these resources and examples:

You can also Google “writing to learn” for more on the subject.

Finally, hot off the press is a report on a multi-year research study of 2,101 writing assignments across 100 higher ed institutions undertaken by Dan Melzer, Associate Professor of English at California State University at Sacramento: Assignments across the Curriculum: A National Study of College Writing, University Press of Colorado, 2014. This is worth taking a look at as you think about what it means to write in specific disciplines and why you might want to integrate writing to learn into your courses.

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image Source: Microsoft Clip Art