Tips for Writing Effective Multiple Choice Questions

Writing test questions is a daunting task for many instructors. It can be challenging to come up with questions that correctly assess students on the comprehension of course objectives. Multiple choice questions are no exception; despite being very popular, instructors often struggle to create well-constructed questions.

Piece of notebook paper with Questions at the top, followed by numbers and ABCD for each of the six numbers. Answers are circled in red.Multiple choice questions have several advantages. They lend themselves to covering a broad range of content and assessing a wide variety of learning objectives. They are very useful when testing a student’s lower level knowledge of a topic, such as factual recall and definitions, but if written correctly, they can be used to assess at the higher levels of analysis, evaluation, and critical thinking skills. Multiple choice questions are scored efficiently (even automatically, if an electronic test is used), therefore, they are frequently the evaluation method preferred by instructors of large courses.

There are some disadvantages, including the fact that this type of question can be time-consuming to construct. Multiple choice questions are made up of two parts: the stem, which identifies the question, and the alternative responses which include the correct answer as well as incorrect alternatives, known as distractors. Coming up with plausible distractors for each question can be a difficult task. And, while some higher level thinking skills can be addressed, multiple choice questions cannot measure a student’s ability to organize and express ideas.  Another thing to consider is that student success when answering multiple choice questions can be influenced by factors unrelated to the subject matter, such as reading ability, deductive reasoning, and the use of context clues.

The following guidelines are offered to help streamline the process of creating multiple choice questions as well as minimize the disadvantages of using them.

General guidelines for writing stems:

  1. When possible, prepare the stem as a clearly written question rather than an incomplete statement.

Poor Example: Psychoanalysis is….

Better example: What is the definition of psychoanalysis? 

  1. Eliminate excessive or irrelevant information from the stem.

Poor example: Jane recently started a new job and can finally afford her own car, a Honda Civic, but is surprised at the high cost of gasoline. Gasoline prices are affected by:

Better example: Which of the following are factors that affect the consumer price of gasoline? 

  1. Include words/phrases in the stem that would otherwise be repeated in the alternatives.

Poor example: Which of the following statements are true?
1. Slowing population growth can prevent global warming
2. Halting deforestation can prevent global warming
3.  Increasing beef production on viable land can prevent global warming
4.  Improving energy efficiency can prevent global warming

Better example: Which of the following techniques can be used to prevent global warming?
1. Slowing population growth
2. Halting deforestation
3. Increasing beef production on viable land
4. Improving energy efficiency 

  1. Avoid using negatively stated stems. If you must use them, highlight the negative word so that it is obvious to students.

Poor example: Which of the following is not a mandatory qualification to be the president of the United States?

Better example: Which of the following is NOT a mandatory qualification to be the president of the United States?

General guidelines for writing alternative responses:

  1. Make sure there is only one correct answer.
  1. Create distractors that are plausible to avoid students guessing the correct answer.

Poor example:
Who was the third president of the United States?
1. George Washington
2. Bugs Bunny
3. Thomas Jefferson
4. Daffy Duck

Better example: Who was the third president of the United States?
1. George Washington
2. Benjamin Franklin
3. Thomas Jefferson
4. John Adams 

  1. Make sure alternative responses are grammatically parallel to each other.

Poor example: Which of the following is the best way to build muscle?
1. Sign up to run a marathon
2. Drinking lots of water
3. Exercise classes
4. Eat protein

Better example: Which of the following is the best way to build muscle?
1. Running on a treadmill
2. Drinking lots of water
3. Lifting weights
4. 
Eating lots of protein 

  1. When possible, list the alternative responses in a logical order (numerical, alphabetical, etc.)

Poor example: How many ounces are in a gallon?
1. 16
2. 148
3. 4
4. 128

Better example: How many ounces are in a gallon?
1. 4
2. 16
3. 128
4. 148

  1. Avoid using ‘All of the above’ or ‘None of the above’ to prevent students from using partial knowledge to arrive at the correct answer.
  2. Use at least four alternative responses to enhance the reliability of the test.

References:

Brame, C., (2013) Writing good multiple choice test questions. Retrieved December 14, 2016 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/

Burton, S. J., Sudweeks, R. R., Merrill, P.F., and Wood, B. (1991). How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. Provo, Utah: Brigham Young University Testing Services and The Department of Instructional Science.

“Multiple Choice Questions.” The University of Texas at Austin Faculty Innovation Center, 14 Dec. 2016, https://facultyinnovate.utexas.edu/teaching/check-learning/question-types/multiple-choice.

Amy Brusini, Blackboard Training Specialist
Center for Educational Resources

Image Source: Pixabay.com

Create an Online Space for Students to Collaborate

Working in groups can be a very positive experience for students; it allows them to take ownership of their learning, and they become active rather than passive learners. In addition to gaining a deeper understanding of the subject at hand, the interaction that students have with their peers is equally valuable. Students have the potential to develop life-long learning skills including critical thinking, problem solving, and decision making abilities, as well as social skills such as effective communication, negotiation, and conflict resolution. Instructors may help ensure the success of group work by following some of these simple guidelines:

  • Establish clear expectations of participation by group members.
  • Specify the roles and responsibilities needed within each group and have students delegate them.
  • Have group members assess each other at various times throughout the project/activity.
  • Periodically check to monitor group progress.
  • Use rubrics to assess both group and individual contributions.

For more on student collaborative work, see Barbara Gross Davis, Tools for Teaching, Jossey-Bass, 2001.)

Screen Shot: Blackboard Groups - Creating a GroupInstructors at JHU who have the option to use Blackboard (course management system), can create an online space for students to work collaboratively. The Groups Tool in Blackboard is a convenient way for instructors to create subsets of students for collaborative activity. Once created, group members have access to a number of communication and collaboration tools within Blackboard, as determined by the instructor. These tools include: a group discussion board, blog, wiki, journal, email tool, chat tool, a file exchange tool, and a task list.  The instructor has access to all group tools at all times.  This allows him/her to monitor each group’s activity as needed. The blog, wiki, and journal tools have the option of being graded; they are connected to the Blackboard grade center, so any grades entered are automatically transferred and recorded in the grade center.  Within these tools there is the option to grade each member of the group individually or grade the group as a whole. Blackboard groups can be created one at a time, or as a set. Members of groups are selected manually, with the instructor choosing students from a pre-populated list, or by using the self-enroll option, where students use a sign-up sheet to enroll themselves into a group. There is also a ‘random enrollment’ feature (if creating a group set), where Blackboard will randomly assign students to groups. This feature is often helpful in large lecture courses.

For more detailed information about using the Blackboard Groups Tool, please see the Groups Tool tutorial.

Amy Brusini, Course Management Training Specialist Center for Educational Resources


Image Source: Screen shot: Blackboard.

2013 GSI Symposium Breakout Session 3: Flipping the Classroom

A Report from the Trenches

We’re continuing with our reports from the JHU Gateway Sciences Initiative (GSI) 2nd Annual Symposium on Excellence in Teaching and Learning in the Sciences. Next up is “Flipping the Classroom: How to Do It Conceptually and Technologically” presented by Michael Falk, Ph.D., Associate Professor of Material Sciences and Engineering  and Brian Cole, Senior Information Technology Specialist, Center for Educational Resources.

Please note that links to examples and explanations in the text below were added by CER staff and were not included in the breakout session presentation.

Instructor with students at computers

For the past several years Professor Michael Falk has “flipped” his course EN.510.202 –Computation and Programming for Materials Scientists and Engineers.  [See the recent Innovative Instructor post on Flipping Your Class.] The purpose of Falk’s class is to teach algorithm development and programming in the context of materials science and engineering.  The class size ranges between 20 and 30 students, and Professor Falk has one Teaching Assistant for the class.

Professor Falk outlined the logistics for the students taking the course. They are required to watch a video of a lecture-style presentation he has posted on his Blackboard course site, and then take a quiz on the content presented in the podcast, before coming to class. The quizzes ensure that the students will watch the lecture and are held accountable for the information presented. Once in class, Falk has the students engage in an interactive experience, such as writing a mini-program, based on the material from the presentation. He noted that he has not found making the podcasts difficult, but creating in-class active learning experiences for his students has been more challenging. He spends a great deal of time developing in-class exercises that will build cumulatively. He also wants students to be able to get enough from the classroom activity to continue work on their own.

For assessment purposes he has students take a survey at the beginning of the semester and at the end of the semester to determine learning gains. Preliminary data indicate that the class increases the ability of students to program, that students showed increased perception in their abilities, as well as an increased intention to use programming in the future.

Brian Cole discussed and demonstrated the technology behind the flipped classroom.  Falk uses the software application ClassSpot, which allows students to share their work on the classroom’s main projection screen, to edit common code during class.  Cole described using Audacity, Adobe Connect, Adobe Presenter, and QuickTime on Macs to create the video recordings.  He mentioned that a faculty member could also use an appropriate pre-recorded lecture from a trusted source. Falk uses ScreenFlow to make his presentations; however, Johns Hopkins does not have a license for this software. Adobe Captivate is another possibility. It is very powerful but has a steeper learning curve.

The follow questions were raised and answered during the session:

Q – Could this method be used to flip a few modules as opposed to the entire course?
A – Undergrads don’t like change, so it would probably be better to do the whole course.

Q – Can students watch the podcasts over and over?
A – Yes.

Q – Where is the textbook in all of this? Could you replace your podcasts with readings from a textbook?
A – There are reading assignments in addition to the videos. In my experience, students prefer a human face, a talking head, over reading a textbook.

Q – How do students reach you if class time is dedicated to working on problems?
A – I encourage students to use the class Blackboard discussion board. [Note: The flipped class structure  doesn’t prevent students from talking to the faculty member, and Falk also has office hours.]

Q – Did you scale back student work [outside of class] since more time spent watching podcasts?
A – Yes – most of the traditional homework is done in class.

Q – Are there tests?
A – Yes.

Q- How important are quizzes to making the flipped course work?
A – Very important. Students are very grade oriented so having quizzes, tests, and exams matters. Quizzes are great motivators for getting students to watch the videos.

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image Source: Microsoft Clip Art

GradeMark Paperless Grading

GradeMark is a paperless grading system that gives instructors the ability to add comments and corrections to assignments submitted electronically. It is a tool offered within Turnitin, the plagiarism detection software product used at JHU. With its drag and drop functionality, among other features, GradeMark has the potential to save instructors a great deal of time when grading online assignments.  It is also easily integrated with Blackboard.

(Note: In order to use GradeMark, online assignments must be created using Turnitin. If using Turnitin within Blackboard, accounts are automatically created for instructors and students through the Blackboard system. If using Turnitin outside of Blackboard, the instructor is responsible for creating separate accounts for each student. Please click here for more information on Turnitin’s integration with Blackboard.)

Screen shot showing example of using GradeMark

GradeMark contains several different grading features:

  • Dragging and Dropping Quickmarks – Quickmarks are frequently used comments that are readily available to drag and drop into a student’s assignment. While viewing an assignment, the instructor can select from a panel of standard Quickmarks that come with GradeMark, or from a custom set that s/he has created.  For example, the abbreviation ‘Awk.’ is a Quickmark indicating an awkward phrase. The ability to drag and drop Quickmarks to an assignment, instead of typing them over and over again, can save instructors a lot of time.
  • General Comments – Each assignment has a generous space where general comments can be added.  General comments can be used to further clarify any Quickmarks that were added as well as discuss the assignment as a whole.
  • Voice Comments – A recent addition to GradeMark is the ability to add voice comments. A voice comment can be added to the assignment lasting up to three minutes in length.  An instructor can use the built-in microphone in his/her computer to easily record the message.
  • Rubrics – Rubrics created within GradeMark can help streamline the grading process by using a ‘scorecard’ approach. Specific criteria and scores are defined in a rubric that is then associated with an assignment. Instructors grade the assignment by filling in the scores based on the evaluative criteria in the rubric. There is also the option of associating Quickmarks with rubrics when they are added to the assignment.

Students are able to view their graded assignments when the ‘post date’ is reached. The post date is set by the instructor when setting up the assignment. Students have the option to print or save a copy of the graded assignment and can view only their own submissions.

GradeMark Logo showing grade book and apple

Advantages:

  • Flexibility in marking up assignments – Quickmarks, rubrics, text, voice comments all available.
  • Time saved dragging and dropping reusable comments.
  • Increased consistency in grading.
  • Clear feedback to students, instead of ‘scribbled margins.’
  • Opportunity to provide more detailed feedback to students including links and resources.
  • No need to download assignments – everything is web-based, stored online.
  • If the instructor is using Blackboard, when the assignment is graded the grade is automatically transferred and recorded into the Blackboard Grade Center.

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image sources: Amy Brusini screen shot of GradeMark example; GradeMark logo

VoiceThread – “Conversations in the Cloud”

VoiceThread is web-based presentation application that allows users to create and share interactive multimedia slideshows. VoiceThread presentations are used to showcase audio, video, images, and documents while allowing users to comment on them in a variety of different ways. Comments can be made using a microphone, a webcam, uploading a prerecorded audio file, using a phone, or by simply typing text. There is also a “doodle” tool which can be used to annotate presentations with digital overlay while leaving a comment.  The result is an ongoing, asynchronous, digital conversation that can be easily shared with individuals, groups, and/or embedded into different websites, including Blackboard.

Image for VoiceThread application. Conversations in the Cloud.

Originally developed at the University of North Carolina, VoiceThread has been used at  the Johns Hopkins Schools of Nursing and Public Health for several years. IT@JH recently obtained a university-wide license for all members of the Hopkins community; instructors and students from all JHU schools now have the ability to access VoiceThread free of charge.

At JHU and other institutions, instructors and students have been very creative in the ways they are using VoiceThread. Here are some examples of how this tool is being used:

  • Student presentation tool – Students can use VoiceThread to create individual or group presentations on any number of topics, which can then be shared with the class.  An added advantage – students can watch and comment on each other’s presentations outside of class, freeing up valuable class time.
  • Online lecture tool – Instructors can use VoiceThread to create online lectures for fully online classes or as a supplement to face-to-face classes.
  • Peer assessment – Students can use VoiceThread to share assignments (papers, images, audio, video clips, etc.) with their peers for comments and critique.
  • Foreign language assessment – VoiceThread is especially useful to foreign language instructors who would like to hear their students speak. Instructors can create a presentation (upload an audio recording, image, video clip, etc.) which students then have to translate, describe, or narrate, for example.
  •  Brainstorming session – Students and instructors can use VoiceThread to brainstorm ideas for project topics, group presentation strategies, etc.
  • Digital storytelling – In groups or independently, students can use VoiceThread to create interactive digital stories using various media artifacts (audio, images, etc.).
  • Review Session – Students can use VoiceThread to record a content review session in preparation for a test or exam.
  • Facilitate Discussions – Students can present a topic and then facilitate a class discussion in VoiceThread about the topic.
  • Student Introductions – Especially helpful in a fully online environment, students and instructors can use VoiceThread to introduce themselves, helping to build a sense of community.

JHU instructors and students can go to http://jhu.voicethread.com and login with their JHED ID and password.  All users are automatically set up with a ‘Basic’ account that they can begin using immediately. There is no software to download as all VoiceThread presentations are created and stored in the “cloud.”

Additional Resources
VoiceThread Overview: https://www.voicethread.com/about/features/
VoiceThread ‘How-To’ Basics: https://www.voicethread.com/support/howto/Basics/
JHSPH VT Site: https://sites.google.com/site/ctltteachingtoolkit/resources/voicethread

Amy Brusini, Course Management Training Specialist
Center for Educational Resources


Image Source: VoiceThread image [http://d25wzyo6b5ic8t.cloudfront.net/rev/c32981bd/media/custom/www/banner_cloud.jpg] edited by Macie Hall