Lunch and Learn: Innovative Grading Strategies

Logo for Lunch and Learn program showing the words Lunch and Learn in orange with a fork above and a pen below the lettering. Faculty Conversations on Teaching at the bottom.On Thursday, February 28, the Center for Educational Resources (CER) hosted the third Lunch and Learn for the 2018-2019 academic year. Rebecca Kelly, Associate Teaching Professor, Earth and Planetary Sciences and Director of the Environmental Science and Studies Program, and Pedro Julian, Associate Professor, Electrical and Computer Engineering, presented on Innovative Grading Strategies.

Rebecca Kelly began the presentation by discussing some of the problems in traditional grading. There is a general lack of clarity in what grades actually mean and how differently they are viewed by students and faculty. Faculty use grades to elicit certain behaviors from students, but it doesn’t necessarily mean that they are learning. Kelly noted that students, especially those at JHU, tend to be focused on the grade itself, aiming for a specific number and not the learning; this often results in high levels of student anxiety, something she sees often. She explained how students here don’t get many chances to fail and not have their grades negatively affected. Therefore, every assessment is a source of stress because it counts toward their grade. There are too few opportunities for students to learn from their mistakes.

Kelly mentioned additional challenges that faculty face when grading: it is often time consuming, energy draining, and stressful, especially when haggling over points, for example.  She makes an effort to provide clearly stated learning goals and rubrics for each assignment, which do help, but are not always enough to ease the burden.

Kelly introduced the audience to specifications grading and described how she’s recently started using this approach in Introduction to Geographic Information Systems (GIS). With specifications grading (also described in a recent CER Innovative Instructor article), students are graded pass/fail or satisfactory/unsatisfactory on individual assessments that align directly with learning goals. Course grades are determined by the number of learning goals mastered. This is measured by the number of assessments passed. For example, passing 20 or more assignments out of 23 would equate to an A; 17-19 assignments would equate to a B. Kelly stresses the importance of maintaining high standards; for rigor, the threshold for passing should be a B or better.

In Kelly’s class, students have multiple opportunities to achieve their goals. Each student receives three tokens that he/she can use to re-do an assignment that doesn’t pass, or select a different assignment altogether from the ‘bundle’ of assignments available. Kelly noted the tendency of students to ‘hoard’ their tokens and how it actually works out favorably; instead of risking having to use a token, students often seek out her feedback before turning anything in.

Introduction to GIS has both a lecture and a lab component. The lab requires students to use software to create maps that are then used to perform data analysis. The very specific nature of the assignments in this class lend themselves well to the specifications grading approach. Kelly noted that students are somewhat anxious about this approach at first, but settle into it once they fully understand. In addition to clearly laying out Grade bundles used in specifications gradingexpectations, Kelly lists the learning goals of the course and how they align with each assignment (see slides). She also provides students with a table showing the bundles of assignments required to reach final course grades. Additionally, she distributes a pacing guide to help students avoid procrastination.

The results that Kelly has experienced with specifications grading have been positive. Students generally like it because the expectations are very clear and initial failure does not count against them; there are multiple opportunities to succeed. Grading is quick and easy because of the pass/fail system; if something doesn’t meet the requirements, it is simply marked unsatisfactory. The quality of student work is high because there is no credit for sloppy work. Kelly acknowledged that specifications grading is not ideal for all courses, but feels the grade earned in her GIS course is a true representation of the student’s skill level in GIS.

Pedro Julian described a different grading practice that he is using, something he calls the “extra grade approach.” He currently uses this approach in Digital Systems Fundamentals, a hands-on design course for freshmen. In this course, Julian uses a typical grading scale: 20% for the midterm, 40% for labs and homework, and 40% for the final project. However, he augments the scale by offering another 20% if students agree to put in extra work throughout the semester. How much extra work? Students must commit to working collaboratively with instructors (and other students seeking the 20% credit) for one hour or more per week on an additional project.  This year, the project is to build a vending machine. Past projects include building an elevator out of Legos and building a robot that followed a specific path on the floor.

Julian described how motivated students are to complete the extra project once they commit to putting in the time. Students quickly realize that they learn all sorts of skills they would not have otherwise learned and are very proud and engaged. Student participation in the “extra grade” option has grown steadily since Julian started using this approach three years ago. The first year there were 5-10 students who signed up, and this year there are 30. Julian showed histograms (see slides) of student grades from past semesters in his class and how the extra grade has helped push overall grades higher.  The histograms also show that it’s not just students who may be struggling with the class who are choosing to participate in the extra grade, but “A students” as well.

Similar to Rebecca Kelly’s experience, Julian expressed how grade-focused JHU students are, much to his dismay. In an attempt to take some of the pressure off, he described how he repeatedly tells his students that if they work hard, they will get a good grade; he even includes this phrase in his syllabus. Julian explained how he truly wants students to concentrate more on the learning and not on the grade, which is his motivation behind the “extra grade” approach.

An interesting discussion with several questions from the audience followed the presentations. Below are some of the questions asked and responses given by Kelly and Julian, as well as audience members.

Q: (for Julian) Some students may not have the time or flexibility in their schedule to take part in an extra project. Do you have suggestions for them? Did you consider this when creating the “extra grade” option?

Julian responded that in his experience, freshmen seem to be available. Many of them make time to come in on the weekends. He wants students to know he’s giving them an “escape route,” a way for them to make up their grade, and they seem to find the time to make it happen.  Julian has never had a student come to him saying he/she cannot participate because of scheduling conflicts.

Q: How has grade distribution changed?

Kelly remarked how motivated the students are and therefore she had no Cs, very few Bs, and the rest As this past semester. She expressed how important it is to make sure that the A is attainable for students. She feels confident that she’s had enough experience to know what counts as an A. Every student can do it, the question is, will they?

Q: (for Kelly) Would there ever be a scenario where students would do the last half of the goals and skip the first half?

Kelly responded that she has never seen anyone jump over everything and that it makes more sense to work sequentially.

Q: (for Kelly) Is there detailed feedback provided when students fail an assignment?

Kelly commented that it depends on the assignment, but if students don’t follow the directions, that’s the feedback – to follow the directions. If it’s a project, Kelly will meet with the student, go over the assignment, and provide immediate feedback. She noted that she finds oral feedback much more effective than written feedback.

Q: (for Kelly) Could specs grading be applied in online classes?

Kelly responded that she thinks this approach could definitely be used in online classes, as long as feedback could be provided effectively. She also stressed the need for rubrics, examples, and clear goals.

Q: Has anyone tried measuring individual learning gains within a class? What skills are students coming in with? Are we actually measuring gain?

Kelly commented that specifications grading works as a compliment to competency based grading, which focuses on measuring gains in very specific skills.

Julian commented that this issue comes up in his class, students coming in with varying degrees of experience. He stated that this is another reason to offer the extra credit, to keep things interesting for those that want to move at a faster pace.

The discussion continued among presenters and audience members about what students are learning in a class vs. what they are bringing in with them. A point was raised that if students already know the material in a class, should they even be there?  Another comment was made regarding if it is even an instructor’s place to determine what students already know.  Additional comments were made about what grades mean and concerns about grades being used for different things, i.e. employers looking for specific skills, instructors writing recommendation letters, etc.

Q: Could these methods be used in group work?

Kelly responded that with specifications grading, you would have to find a way to evaluate the group. It might be possible to still score on an individual basis within the group, but it would depend on the goals. She mentioned peer evaluations as a possibility.

Julian stated that all grades are based on individual work in his class. He does use groups in a senior level class that he teaches, but students are still graded individually.

The event concluded with a discussion about how using “curve balls” – intentionally difficult questions designed to catch students off-guard – on exams can lead to challenging grading situations. For example, to ultimately solve a problem, students would need to first select the correct tools before beginning the solution process. Some faculty were in favor of including this type of question on exams, while others were not, noting the already high levels of exam stress.  A suggestion was made to give students partial credit for the process even if they don’t end up with the correct answer. Another suggestion was to give an oral exam in order to hear the student’s thought process as he/she worked through the challenge. This would be another way for students to receive partial credit for their ideas and effort, even if the final answer was incorrect.

Amy Brusini, Senior Instructional Designer
Center for Educational Resources

Image Sources: Lunch and Learn Logo, slide from Kelly presentation

Changing the Guard

After 31 years at Johns Hopkins University, 11 in the Center for Educational Resources, six years and 198 posts on The Innovative Instructor, I am retiring. The good news is that my colleague, Amy Brusini, will be taking on the mantle, not only as the new editor of this blog, but as the Senior Instructional Designer in the CER.

Amy has been an Instructional Technology Specialist in the CER for 12 years. She has a BS in Music Education from Towson University and a MS in Education from Johns Hopkins University. She is currently the CER’s Blackboard expert, providing instruction and consultations to faculty for course management, instructional design, and educational technology.

The Innovative Instructor has been a labor of love for me. I am leaving it in good hands. Happy Trails!

Macie Hall, Senior Instructional Designer
Center for Educational Resources

Image Sources: CC Reid Sczerba, Center for Educational Resources

Tips for Writing Effective Multiple Choice Questions

Writing test questions is a daunting task for many instructors. It can be challenging to come up with questions that correctly assess students on the comprehension of course objectives. Multiple choice questions are no exception; despite being very popular, instructors often struggle to create well-constructed questions.

Piece of notebook paper with Questions at the top, followed by numbers and ABCD for each of the six numbers. Answers are circled in red.Multiple choice questions have several advantages. They lend themselves to covering a broad range of content and assessing a wide variety of learning objectives. They are very useful when testing a student’s lower level knowledge of a topic, such as factual recall and definitions, but if written correctly, they can be used to assess at the higher levels of analysis, evaluation, and critical thinking skills. Multiple choice questions are scored efficiently (even automatically, if an electronic test is used), therefore, they are frequently the evaluation method preferred by instructors of large courses.

There are some disadvantages, including the fact that this type of question can be time-consuming to construct. Multiple choice questions are made up of two parts: the stem, which identifies the question, and the alternative responses which include the correct answer as well as incorrect alternatives, known as distractors. Coming up with plausible distractors for each question can be a difficult task. And, while some higher level thinking skills can be addressed, multiple choice questions cannot measure a student’s ability to organize and express ideas.  Another thing to consider is that student success when answering multiple choice questions can be influenced by factors unrelated to the subject matter, such as reading ability, deductive reasoning, and the use of context clues.

The following guidelines are offered to help streamline the process of creating multiple choice questions as well as minimize the disadvantages of using them.

General guidelines for writing stems:

  1. When possible, prepare the stem as a clearly written question rather than an incomplete statement.

Poor Example: Psychoanalysis is….

Better example: What is the definition of psychoanalysis? 

  1. Eliminate excessive or irrelevant information from the stem.

Poor example: Jane recently started a new job and can finally afford her own car, a Honda Civic, but is surprised at the high cost of gasoline. Gasoline prices are affected by:

Better example: Which of the following are factors that affect the consumer price of gasoline? 

  1. Include words/phrases in the stem that would otherwise be repeated in the alternatives.

Poor example: Which of the following statements are true?
1. Slowing population growth can prevent global warming
2. Halting deforestation can prevent global warming
3.  Increasing beef production on viable land can prevent global warming
4.  Improving energy efficiency can prevent global warming

Better example: Which of the following techniques can be used to prevent global warming?
1. Slowing population growth
2. Halting deforestation
3. Increasing beef production on viable land
4. Improving energy efficiency 

  1. Avoid using negatively stated stems. If you must use them, highlight the negative word so that it is obvious to students.

Poor example: Which of the following is not a mandatory qualification to be the president of the United States?

Better example: Which of the following is NOT a mandatory qualification to be the president of the United States?

General guidelines for writing alternative responses:

  1. Make sure there is only one correct answer.
  1. Create distractors that are plausible to avoid students guessing the correct answer.

Poor example:
Who was the third president of the United States?
1. George Washington
2. Bugs Bunny
3. Thomas Jefferson
4. Daffy Duck

Better example: Who was the third president of the United States?
1. George Washington
2. Benjamin Franklin
3. Thomas Jefferson
4. John Adams 

  1. Make sure alternative responses are grammatically parallel to each other.

Poor example: Which of the following is the best way to build muscle?
1. Sign up to run a marathon
2. Drinking lots of water
3. Exercise classes
4. Eat protein

Better example: Which of the following is the best way to build muscle?
1. Running on a treadmill
2. Drinking lots of water
3. Lifting weights
4. 
Eating lots of protein 

  1. When possible, list the alternative responses in a logical order (numerical, alphabetical, etc.)

Poor example: How many ounces are in a gallon?
1. 16
2. 148
3. 4
4. 128

Better example: How many ounces are in a gallon?
1. 4
2. 16
3. 128
4. 148

  1. Avoid using ‘All of the above’ or ‘None of the above’ to prevent students from using partial knowledge to arrive at the correct answer.
  2. Use at least four alternative responses to enhance the reliability of the test.

References:

Brame, C., (2013) Writing good multiple choice test questions. Retrieved December 14, 2016 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/

Burton, S. J., Sudweeks, R. R., Merrill, P.F., and Wood, B. (1991). How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. Provo, Utah: Brigham Young University Testing Services and The Department of Instructional Science.

“Multiple Choice Questions.” The University of Texas at Austin Faculty Innovation Center, 14 Dec. 2016, https://facultyinnovate.utexas.edu/teaching/check-learning/question-types/multiple-choice.

Amy Brusini, Blackboard Training Specialist
Center for Educational Resources

Image Source: Pixabay.com

Create an Online Space for Students to Collaborate

Working in groups can be a very positive experience for students; it allows them to take ownership of their learning, and they become active rather than passive learners. In addition to gaining a deeper understanding of the subject at hand, the interaction that students have with their peers is equally valuable. Students have the potential to develop life-long learning skills including critical thinking, problem solving, and decision making abilities, as well as social skills such as effective communication, negotiation, and conflict resolution. Instructors may help ensure the success of group work by following some of these simple guidelines:

  • Establish clear expectations of participation by group members.
  • Specify the roles and responsibilities needed within each group and have students delegate them.
  • Have group members assess each other at various times throughout the project/activity.
  • Periodically check to monitor group progress.
  • Use rubrics to assess both group and individual contributions.

For more on student collaborative work, see Barbara Gross Davis, Tools for Teaching, Jossey-Bass, 2001.)

Screen Shot: Blackboard Groups - Creating a GroupInstructors at JHU who have the option to use Blackboard (course management system), can create an online space for students to work collaboratively. The Groups Tool in Blackboard is a convenient way for instructors to create subsets of students for collaborative activity. Once created, group members have access to a number of communication and collaboration tools within Blackboard, as determined by the instructor. These tools include: a group discussion board, blog, wiki, journal, email tool, chat tool, a file exchange tool, and a task list.  The instructor has access to all group tools at all times.  This allows him/her to monitor each group’s activity as needed. The blog, wiki, and journal tools have the option of being graded; they are connected to the Blackboard grade center, so any grades entered are automatically transferred and recorded in the grade center.  Within these tools there is the option to grade each member of the group individually or grade the group as a whole. Blackboard groups can be created one at a time, or as a set. Members of groups are selected manually, with the instructor choosing students from a pre-populated list, or by using the self-enroll option, where students use a sign-up sheet to enroll themselves into a group. There is also a ‘random enrollment’ feature (if creating a group set), where Blackboard will randomly assign students to groups. This feature is often helpful in large lecture courses.

For more detailed information about using the Blackboard Groups Tool, please see the Groups Tool tutorial.

Amy Brusini, Course Management Training Specialist Center for Educational Resources


Image Source: Screen shot: Blackboard.

2013 GSI Symposium Breakout Session 3: Flipping the Classroom

A Report from the Trenches

We’re continuing with our reports from the JHU Gateway Sciences Initiative (GSI) 2nd Annual Symposium on Excellence in Teaching and Learning in the Sciences. Next up is “Flipping the Classroom: How to Do It Conceptually and Technologically” presented by Michael Falk, Ph.D., Associate Professor of Material Sciences and Engineering  and Brian Cole, Senior Information Technology Specialist, Center for Educational Resources.

Please note that links to examples and explanations in the text below were added by CER staff and were not included in the breakout session presentation.

Instructor with students at computers

For the past several years Professor Michael Falk has “flipped” his course EN.510.202 –Computation and Programming for Materials Scientists and Engineers.  [See the recent Innovative Instructor post on Flipping Your Class.] The purpose of Falk’s class is to teach algorithm development and programming in the context of materials science and engineering.  The class size ranges between 20 and 30 students, and Professor Falk has one Teaching Assistant for the class.

Professor Falk outlined the logistics for the students taking the course. They are required to watch a video of a lecture-style presentation he has posted on his Blackboard course site, and then take a quiz on the content presented in the podcast, before coming to class. The quizzes ensure that the students will watch the lecture and are held accountable for the information presented. Once in class, Falk has the students engage in an interactive experience, such as writing a mini-program, based on the material from the presentation. He noted that he has not found making the podcasts difficult, but creating in-class active learning experiences for his students has been more challenging. He spends a great deal of time developing in-class exercises that will build cumulatively. He also wants students to be able to get enough from the classroom activity to continue work on their own.

For assessment purposes he has students take a survey at the beginning of the semester and at the end of the semester to determine learning gains. Preliminary data indicate that the class increases the ability of students to program, that students showed increased perception in their abilities, as well as an increased intention to use programming in the future.

Brian Cole discussed and demonstrated the technology behind the flipped classroom.  Falk uses the software application ClassSpot, which allows students to share their work on the classroom’s main projection screen, to edit common code during class.  Cole described using Audacity, Adobe Connect, Adobe Presenter, and QuickTime on Macs to create the video recordings.  He mentioned that a faculty member could also use an appropriate pre-recorded lecture from a trusted source. Falk uses ScreenFlow to make his presentations; however, Johns Hopkins does not have a license for this software. Adobe Captivate is another possibility. It is very powerful but has a steeper learning curve.

The follow questions were raised and answered during the session:

Q – Could this method be used to flip a few modules as opposed to the entire course?
A – Undergrads don’t like change, so it would probably be better to do the whole course.

Q – Can students watch the podcasts over and over?
A – Yes.

Q – Where is the textbook in all of this? Could you replace your podcasts with readings from a textbook?
A – There are reading assignments in addition to the videos. In my experience, students prefer a human face, a talking head, over reading a textbook.

Q – How do students reach you if class time is dedicated to working on problems?
A – I encourage students to use the class Blackboard discussion board. [Note: The flipped class structure  doesn’t prevent students from talking to the faculty member, and Falk also has office hours.]

Q – Did you scale back student work [outside of class] since more time spent watching podcasts?
A – Yes – most of the traditional homework is done in class.

Q – Are there tests?
A – Yes.

Q- How important are quizzes to making the flipped course work?
A – Very important. Students are very grade oriented so having quizzes, tests, and exams matters. Quizzes are great motivators for getting students to watch the videos.

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image Source: Microsoft Clip Art

GradeMark Paperless Grading

GradeMark is a paperless grading system that gives instructors the ability to add comments and corrections to assignments submitted electronically. It is a tool offered within Turnitin, the plagiarism detection software product used at JHU. With its drag and drop functionality, among other features, GradeMark has the potential to save instructors a great deal of time when grading online assignments.  It is also easily integrated with Blackboard.

(Note: In order to use GradeMark, online assignments must be created using Turnitin. If using Turnitin within Blackboard, accounts are automatically created for instructors and students through the Blackboard system. If using Turnitin outside of Blackboard, the instructor is responsible for creating separate accounts for each student. Please click here for more information on Turnitin’s integration with Blackboard.)

Screen shot showing example of using GradeMark

GradeMark contains several different grading features:

  • Dragging and Dropping Quickmarks – Quickmarks are frequently used comments that are readily available to drag and drop into a student’s assignment. While viewing an assignment, the instructor can select from a panel of standard Quickmarks that come with GradeMark, or from a custom set that s/he has created.  For example, the abbreviation ‘Awk.’ is a Quickmark indicating an awkward phrase. The ability to drag and drop Quickmarks to an assignment, instead of typing them over and over again, can save instructors a lot of time.
  • General Comments – Each assignment has a generous space where general comments can be added.  General comments can be used to further clarify any Quickmarks that were added as well as discuss the assignment as a whole.
  • Voice Comments – A recent addition to GradeMark is the ability to add voice comments. A voice comment can be added to the assignment lasting up to three minutes in length.  An instructor can use the built-in microphone in his/her computer to easily record the message.
  • Rubrics – Rubrics created within GradeMark can help streamline the grading process by using a ‘scorecard’ approach. Specific criteria and scores are defined in a rubric that is then associated with an assignment. Instructors grade the assignment by filling in the scores based on the evaluative criteria in the rubric. There is also the option of associating Quickmarks with rubrics when they are added to the assignment.

Students are able to view their graded assignments when the ‘post date’ is reached. The post date is set by the instructor when setting up the assignment. Students have the option to print or save a copy of the graded assignment and can view only their own submissions.

GradeMark Logo showing grade book and apple

Advantages:

  • Flexibility in marking up assignments – Quickmarks, rubrics, text, voice comments all available.
  • Time saved dragging and dropping reusable comments.
  • Increased consistency in grading.
  • Clear feedback to students, instead of ‘scribbled margins.’
  • Opportunity to provide more detailed feedback to students including links and resources.
  • No need to download assignments – everything is web-based, stored online.
  • If the instructor is using Blackboard, when the assignment is graded the grade is automatically transferred and recorded into the Blackboard Grade Center.

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image sources: Amy Brusini screen shot of GradeMark example; GradeMark logo

VoiceThread – “Conversations in the Cloud”

VoiceThread is web-based presentation application that allows users to create and share interactive multimedia slideshows. VoiceThread presentations are used to showcase audio, video, images, and documents while allowing users to comment on them in a variety of different ways. Comments can be made using a microphone, a webcam, uploading a prerecorded audio file, using a phone, or by simply typing text. There is also a “doodle” tool which can be used to annotate presentations with digital overlay while leaving a comment.  The result is an ongoing, asynchronous, digital conversation that can be easily shared with individuals, groups, and/or embedded into different websites, including Blackboard.

Image for VoiceThread application. Conversations in the Cloud.

Originally developed at the University of North Carolina, VoiceThread has been used at  the Johns Hopkins Schools of Nursing and Public Health for several years. IT@JH recently obtained a university-wide license for all members of the Hopkins community; instructors and students from all JHU schools now have the ability to access VoiceThread free of charge.

At JHU and other institutions, instructors and students have been very creative in the ways they are using VoiceThread. Here are some examples of how this tool is being used:

  • Student presentation tool – Students can use VoiceThread to create individual or group presentations on any number of topics, which can then be shared with the class.  An added advantage – students can watch and comment on each other’s presentations outside of class, freeing up valuable class time.
  • Online lecture tool – Instructors can use VoiceThread to create online lectures for fully online classes or as a supplement to face-to-face classes.
  • Peer assessment – Students can use VoiceThread to share assignments (papers, images, audio, video clips, etc.) with their peers for comments and critique.
  • Foreign language assessment – VoiceThread is especially useful to foreign language instructors who would like to hear their students speak. Instructors can create a presentation (upload an audio recording, image, video clip, etc.) which students then have to translate, describe, or narrate, for example.
  •  Brainstorming session – Students and instructors can use VoiceThread to brainstorm ideas for project topics, group presentation strategies, etc.
  • Digital storytelling – In groups or independently, students can use VoiceThread to create interactive digital stories using various media artifacts (audio, images, etc.).
  • Review Session – Students can use VoiceThread to record a content review session in preparation for a test or exam.
  • Facilitate Discussions – Students can present a topic and then facilitate a class discussion in VoiceThread about the topic.
  • Student Introductions – Especially helpful in a fully online environment, students and instructors can use VoiceThread to introduce themselves, helping to build a sense of community.

JHU instructors and students can go to http://jhu.voicethread.com and login with their JHED ID and password.  All users are automatically set up with a ‘Basic’ account that they can begin using immediately. There is no software to download as all VoiceThread presentations are created and stored in the “cloud.”

Additional Resources
VoiceThread Overview: https://www.voicethread.com/about/features/
VoiceThread ‘How-To’ Basics: https://www.voicethread.com/support/howto/Basics/
JHSPH VT Site: https://sites.google.com/site/ctltteachingtoolkit/resources/voicethread

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image Source: VoiceThread image [http://d25wzyo6b5ic8t.cloudfront.net/rev/c32981bd/media/custom/www/banner_cloud.jpg] edited by Macie Hall