September 26, 2007
Updated on: November 27th, 2007
One thing that has greatly disappointed me about college undergrad courses is the reliance many professors place on multiple choice (MC) exams. This semester, all five of the classes I'm taking have used multiple choice questions on exams. This is unfortunate because, as I will attempt to make the case, multiple choice exams have many flaws and are a poor metric to determine a student's knowledge on a given subject. Many of the arguments here have been borrowed from the National Center for Fair and Open Testing and examples are from my own experiences with multiple choice testing.
Below are my objections to multiple choice exams:
MC exam results are mathematically skewed - say I sit down at a 500 question multiple choice exam for a class I never attended, never read the textbook, and know nothing about: when I receive the grade for my exam, I'll have received a score of 25% (within some standard deviation) if I filled in all the bubbles randomly. Clearly, I do not understand 25% of the material, I understand 0% of it. My score significantly misrepresents my knowledge on the topic. Of course, this only works if all bubbles are filled in randomly, but there are other mathematical problems with multiple choice...
MC exam questions are all or nothing - since every question only has one right answer, I can receive 0% (completely failure) for a question, even if I have some amount of knowledge on the subject. Consider the following example question: Q: Was the infantry invasion of Japan a viable alternative to the use of the atomic bomb to end World War II? Is so, why? If not, why not?
A. Yes; transport ships were available in sufficient numbers.
B. Yes; island defenses in Japan were minimal.
C. No; estimated casualties would have been much greater.
D. No; Japan was on the verge of having an atomic bomb.
Lets say I think the answer is "No" and I can make a great case to justify that answer. Perhaps I would argue that Japan's military was more powerful than the U.S. believed and that our military would have lost in an infantry invasion. Unfortunately, that isn't a choice; and now I am forced to guess between answers C and D. The "wanted answer" in this example is C. Had I guessed D, I would have received 0% credit and failed the question, even though I had a strong grasp of the subject; statistically, I have a 50% chance of getting the answer correct, which is unfair if I understand, say, 80% of the subject.
Mutiple choice exams often employ misleading or poorly worded questions - rather than ask a question and allow the student to answer it as he/she chooses, MC exams force students to try to decipher the meaning of a question and think "what does the professor want me to say". Consider this example from a recent psychology exam I took: Q: Smoking during pregnancy is most likely to cause which of the following birth defects?
A. Low birth weight
C. General Anxiety
D. Smoking will cause many birth defects
I look at this question and think... "well, the most common birth defect from smoking is low birth weight, BUT I know that smoking causes other birth defects as well." So now I'm torn over whether I should pick A or D. I know that smoking causes a number of birth defects, including preterm birth, cerebral palsy, mental retardation, learning problems, withdrawal symptoms, etc. So even though I know more about this subject than the question is asking, my goal is to decipher what the professor means by the terms "most" and "many". Some of my professors in the past have used "most" to refer to one or more correct answer, but this particular professor wanted it to mean "the number one statistical birth defect, excluding the number two, three, etc. defects". I have also had professors who use the word "many" as an inclusive term; so to me, so I interpret choice D as "low birth weight, preterm birth, cerebral palsy, etc." I chose answer D and received 0% (complete failure) for the question because I thought too deeply into the question and misjudged the intent of the professor. Had this been a short answer question, I could have identified that low birth weight is the statistically most common birth defect, but included that there is still a risk of developing other problems. Since it was multiple choice, I failed the question and have no way to prove what I was thinking at the time.
Multiple choice exams discourage learning outside of a narrow curriculum - when studying for multiple choice exams, I often find myself memorizing vocabulary terms or cramming in concepts I know will be on the exam. Rarely do I look at the big picture, and often I forget the material hours or days after the exam ends. The very nature of MC exams discourages learning broad concepts because you are often punished (as in the above example) for knowing too much. On a multiple choice economics exam, for example, it is more important to memorize the metrics used to measure GDP than it is to know the connections between GDP and other important aspects of economics. Critical thinking is discouraged; mindless memorization is the priority.
The concept of a "best answer" makes MC questions confusing and frustrating - MC questions often have many choices that could be "plausible" answers but they force students to choose the "best answer." This is problematic because "best" answer is a vague concept and because choosing the "second best" answer might still demonstrate some understanding of a concept. Unfortunately, since multiple choice is all or nothing, this often creates extreme frustration for students. Consider this example from a recent philosophy exam I took: Which of the following can be known using a posteriori knowledge (a posteriori = empirical evidence or use of human senses):
A. Triangle have three sides
B. Martians live on Mars
C. All bachelors are unmarried
D. Descartes wrote Meditations on First Philosophy
At first glance I choose B, because obviously if there was life on Mars I would have to see pictures with my eyes or hear about it from someplace. But wait, choice D also seems correct, because in order to know what Descartes authored I'd have to read his works or learn about it from someplace else. So whats going on? Apparently there are two correct answers here. I pick B and am wrong. The professor says the answer is D because Descartes really did write Meditations on First Philosophy but there is no known life on Mars. After trying to argue my case I'm told that even though B might be a plausible answer, D is the "best" answer and that I should have picked D. Forget the fact that B is a plausible answer and forget the fact that I understand the concept of a posteriori knowledge and can justify my answer... I get 0% and fail the question.
"All of the above" and "none of the above" remove the "choice" from multiple choice - my Psychology professor justifies multiple choice exams by stating that they are easier than any other exam formats because they require the tester only to "recognize" the correct answer from a list of choices rather than recall some information. To understand this theory, think of it this way: say you've watched a few football games in your life but know virtually nothing about the game. If someone asks you what it is called when a player runs the ball into the end zone, you would probably have no idea. But what if you could choose between the following choices: run, basket, touchdown, goal. Ah ha! Now you remember that its called a touchdown. So isn't that proof that multiple choice is easier? Maybe for very basic questions, but not when you throw in "all of the above" or "none of the above" answers into the mix. Consider this example from the same professor's exam: If you occasionally attend a rock concert without wearing ear protection, chances are good that you would suffer:
A. Total hearing loss after a few years
B. Damage to hair cells
C. Damage to eardrum
D. All of the above
This question suffers from many of the downfalls of multiple choice exams. First, it uses many misleading words, like "occasionally", "good" and "few". Second, it relies on a "best answer" since A, B and C are all theoretically plausible. And finally, it offers a choice to choose "all of the above." This question goes way beyond simply recognizing the correct answer, it can only be solved by playing mind games and trying to figure out what the professor wants you to say. I pick choice D, figuring that since all of the above are plausible answers, chances would be "good" that all of them would occur. Plus I figure that since the D choice is inclusive of the best choice, it would then become the best choice. I'm wrong (again) since there isn't a "good" chance of damaging the eardrum or developing total hearing loss (even after a "few" years) and therefore the answer which includes those choices is also wrong.
True/False exams are multiple choice in disguise - some people believe that true/false is the easiest style of testing because for every question there is a 50% chance of getting it correct, even without knowing the material! Fair enough; but when you "do" know the material, these questions can be a royal pain, since its often possible to justify both the true and the false choices in your head. Take this example from a recent management exam I took: True or false: honesty is absolutely essential to leadership? The obvious answer to the question would be "true" but for a few seconds I might contemplate a scenario where honesty would not be essential to leadership. After all, if there is just one exception to that statement then the "false" answer would seem plausible, making this a matter of picking which is better, a tricky task.
Multiple Choice is not more "objective" than other exam formats when applied to a single class - one of the arguments some people use to justify multiple choice exams is that they can be graded more objectively than other forms of testing. For example, on the SAT, the essay section is graded by hundreds of different people across the country... in theory your score could be completely different depending on who is reading your exam. You might write a very similar essay to the person sitting next to you during the exam but she might get great marks and you might not do well at all. On the other hand, multiple choice, which is graded by a machine, can't succumb to such subjectivity. While I give some credence to this argument for huge nation-wide standardized tests, the argument fails to hold up when applied to exams in individual classes. Since one professor will be grading all the exams, there is little risk of grading discrepancy between individual students.
So with all the problems multiple choice exams have, why do professors use them? One reason I already noted above, they have the faulty belief that multiple choice relies on "recognition" rather than "recall" and therefore are easier for students; but this line of thought seems fairly rare. In many instances, it turns out that professors use multiple choice because students actually prefer them to any other test format. How can this be? I thought about it for a few days and asked people who prefer multiple choice exams and came up with a theory.
All of the reasons I am opposed to multiple choice exams are predicated on the assumption that I understand the material and that multiple choice exams hinder me from demonstrating what I know. If, however, I didn't know the material, or didn't know it well, then I might be led to believe that I'd be better off with a multiple choice exam. After all, with multiple choice exams, at least there would be a chance I could "recognize" something and at least I have something to pick from. On a short answer exam, if I don't know the material I'll end up sitting there staring at blank sheets of paper for the entire exam period. Psychologically I think being able to pick an answer to a bunch of questions (even if the answers are wrong) is more comforting than having a bunch a bunch of blank spaces. A 50% is a 50%, regardless of whether you filled in all the bubbles and missed half or if you had a bunch of blank space on a short answer test. But while taking the test, the student will feel more confident filling in the bubbles than anything else.
Does this mean professors should still use multiple choice if a lot of students prefer it? I don't think so. The ultimate goal of any professor should be to teach as much useful material as possible and the ultimate goal of students should be to learn as much as possible. As I hope I've demonstrated, multiple choice exams do a poor job helping students learn important material but instead encourage them to "beat the test". If professors truly care about teaching students material that they can use in the future, they should shred their multiple choice exams and not look back.
After all this negativity about multiple choice, do I have a solution? Well, nothing is perfect and no one will ever be fully satisfied, but there are two exam styles which are both fair and encourage students to learn and retain material. The first consists of a series of short answer questions where students are instructed to answer some percentage of them. For example, the professor could give 5 questions and ask students to answer 4 or give 10 questions and ask students to answer 7. This style of exam ensures that students have a fair opportunity to demonstrate what they know about a topic and also gives students a confidence boost, knowing that they only have to answer those questions they know the most about.
The second style of testing is rare but seems both fair and reasonable. The professor will hand out several short answer questions 3-7 days before an exam, and 70%-80% of them will appear on the actual exam. This guarantees that students study the concepts that professors deem to be most important; the element of randomness ensures that the student prepares for all the questions, since they don't know which will be on the exam, preparing for only some would be nothing more than a roll of the dice; and having the questions beforehand gives students confidence and relieves some test anxiety.
I think if more students had the opportunity to experience these types of exams they would become increasingly negative on multiple choice. I think multiple choice could seem attractive when compared to short answer exams where students have to answer every single question; but I think the two exam styles I've proposed can solve many of the problems students currently face. Ultimately, nothing will be perfect, but we can do a better job of making sure that expensive college courses are worth it - alternatives to multiple choice exams can help.