Writing Multiple Choice Questions to assess Higher Order Thinking Skills

There is a common misconception that multiple choice questions (MCQs) can only be used to assess lower level cognitive abilities, such as Remembering and Understanding [1]. However, when appropriately designed, MCQs can assess more complex abilities, such as Applying and Analysing. MCQs are regularly adopted for assessment purposes given their dual benefits of automated marking and standardisation. If you are considering using MCQs to assess more complex cognitive abilities, this deep dive resource will help you answer the following questions:

  • Are MCQs a suitable choice for your assessment?
  • To what extent can MCQs assess higher order thinking skills (HOTS)?
  • How can you write MCQs to assess HOTS?

Theme 1: The Suitability of MCQs

This theme explores the benefits and limitations of MCQs, as well as proposing some key questions to help you decide whether MCQs are an appropriate method for your assessment.

  • MCQs address contextual assessment challenges, such as large cohort sizes and teacher marking workload. While they require more initial time to develop, they subsequently require less resource [6, 11].
  • MCQs reduce bias and results are more objective compared to other forms of assessment, such as essays [6].
  • MCQs can provide increased student flexibility since online tests can be done at a time and place convenient to the student [6].

  • While MCQs have several practical benefits, their adoption is not always driven by pedagogy. Therefore, sometimes they are adopted despite not being the ‘ideal’ form of assessment [6, 11].
  • MCQs are better for assessment of learning than assessment for learning, meaning they can be a useful tool for measuring what learning has happened but do not necessarily promote student learning itself [11].
  • There are ongoing debates over the extent to which MCQs can assess more complex cognitive abilities [1, 7, 11, 14]. This will be discussed further in theme 2.
  • Feedback given on MCQs is often limited and predetermined [6].

Before deciding whether to adopt MCQs for your assessment, consider the following:

  • What might be some of the benefits of using MCQs in your context?
  • What might be some of the limitations of using MCQs in your context?

Theme 1 takeaways

Choose MCQs because they are the right assessment method, not because they are convenient.

The benefits of MCQsThe limitations of MCQs
MCQs can reduce teacher marking workloadMCQs require upfront development time
MCQs can address challenges associated with assessing larger cohortsMCQs are sometimes chosen for practical reasons rather than pedagogical reasons
MCQs can reduce marking bias and subjectivityMCQs may not assess higher cognitive abilities
MCQs can provide increased student flexibilityMCQs feedback is often limited and predetermined

Theme 2: Can MCQs assess HOTS?

This theme explores the extent to which MCQs can be used to assess higher order thinking skills (HOTS).

Bloom’s Taxonomy (1956) is widely used by educators to refer to the different types of cognitive processing involved in learning which are hierarchical in their complexity, ranging from ‘lower-order’ to ‘higher-order’. In the revised version of Bloom’s Taxonomy [1], the levels of cognitive processing are:

  • Remember (lower-order)
  • Understand (lower-order)
  • Apply (higher-order)
  • Analyse (higher-order)
  • Evaluate (higher-order)
  • Create (higher-order)
Adapted from Revised Bloom’s Taxonomy (Anderson & Krathwohl, 2001)

Higher-order thinking skills (HOTS) go beyond simply memorising and comprehending information, to more complex problem solving and the generation of new knowledge.

Bloom’s Taxonomy can be a useful tool to refer to when designing MCQs [10].

Overall, sources tend to agree that HOTS can be assessed using MCQs, but exclude the two highest levels, Evaluating and Creating [1, 7, 14, 15].

  • Analysis of Medical College Admission Test (MCAT), which was comprised only of MCQs, excluded the top two higher levels of cognition in Bloom’s taxonomy, leaving only ‘application’ and ‘analysis’ being assessed by MCQs [15].
  • In a study by Lui et al. (2024), academic staff interviewed agreed the highest two levels of Blooms could not be reached with MCQs.
  • In practice, MCQ questions often mainly test the bottom two levels of Bloom’s Taxonomy, Remembering and Understanding [11].

In planning your assessment strategy, you should consider how the Intended Learning Outcomes (ILOs) for your course are being measured across all your assessments [2]. Douglas (2012) stresses the importance of taking a holistic approach to designing formative and summative assessments and highlights that MCQs can be an effective and reliable method of measuring comprehension levels when paired with other methods.

In preparing to use MCQs to assess HOTS, consider the following:

  • What are the Intended Learning Outcomes (ILOs) for your course?
  • With which level(s) of cognitive ability do your ILOs align? Are those levels assessable using MCQs?
  • What other formative and summative assessments will you include to complement using MCQs?

Theme 2 takeaways

  • MCQs can be used to assess HOTS (Applying and Analysing), but not the highest levels (Evaluating and Creating).
  • In practice, MCQs often don’t assess HOTS. If the goal is to assess HOTS, MCQs must be thoughtfully and intentionally designed to do so.

Theme 3: Writing Effective MCQs to assess HOTS

This theme discusses how to effectively write MCQs that assess HOTS, including some general tips on writing good MCQ questions.

Before we consider writing MCQs for HOTS, let’s take a look at some general tips from Haladyna et al. (2002)’s A Revised Taxonomy of Multiple-Choice (MC) Item-Writing Guidelines (access A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment for the full guidelines):

  • Focus on ILOs: Write questions that assess the key knowledge and skills students are expected to demonstrate, rather than testing trivial facts or isolated details. 
  • Write a clear, focused question: Ensure the stem presents a single, well-defined problem or question. Keep wording concise and include all essential information in the stem rather than the answer options. 
  • Use clear and accessible language: Avoid unnecessary complexity, excessive reading, trick questions, and ambiguous wording so that students are assessed on the ILOs, not their ability to interpret the question. 
  • Provide one best answer with plausible distractors: Include a single clearly correct answer alongside distractors that are realistic and reflect common misconceptions or errors. 
  • Avoid unintended clues: Keep answer options similar in length, style, and grammatical structure, and avoid patterns or wording that might reveal the correct answer without students demonstrating understanding. 
  • Review and test your questions: Proofread carefully and, where possible, pilot questions or review student responses to identify ambiguity, poor distractors, or questions that do not perform as intended.

Writing MCQs to assess HOTS

When it comes to assessing HOTS, here are some methods you can explore and examples to draw upon to create questions that test more complex cognitive processes, such as Applying and Analysing.

To avoid simply recalling information, “[use] novel material to test higher level learning” [8]. Paraphrase language used during instruction to construct new examples in which students can apply their knowledge.

Take a look at the following example from Haataja et al. (2023, p.6):

Students read source material and then answer questions.

Question: Here are statements on students’ undesired behavior in lessons. Associate each statement to the model or theory which it represents the most clearly in its means of explaining.

  1. Hempel’s model,
  2. Contrafactual theory,
  3. Interventionistic theory,
  4. None of the aforementioned

Statement 1: If a teacher notices that the students disagree on the desirable behavior with her, she should guide the students to negotiations on shared values

Another method of ensuring MCQs assess more complex cognitive processes is by having students answer questions in which they must consider multiple sources of information to reach the correct solution.

Take a look at the following example from Haataja et al. (2023, p.6/7):

Students read two research papers, one empirical, one theoretical.

Question: Which of the following items is correct according to the papers?

  1. Peltola et al.‘s study [the empirical paper] adopts the approach of positivistic research tradition [the theoretical paper].
  2. The Standpoint of the Subject [the empirical paper] is a special example of research that explains human behavior with covering law model [the theoretical paper].

Real-world scenarios (also known as ‘vignettes’) can be useful in getting students to demonstrate HOTS [12], as well as source materials, such as patient charts, report excerpts, and data visualisations. Using scenarios and source materials can enhances the authenticity of the assessment and moves students from simply understanding the facts to applying them within a real-world context. Questions such as these also require students to differentiate between relevant and irrelevant information. Media, such as images, video, and sound files, can be included in scenarios/materials to enhance authenticity and text skills beyond only text-based [12].

Take a look at the following example of a vignette from the National Board of Medical Examiners (NBME, 2012, p.38):

“A 47-year-old woman comes to the emergency department because of shortness of breath and left lower extremity pain with ambulation. Yesterday she returned from Europe after a 10-hour flight. She has no remarkable medical history and takes only an oral contraceptive daily. Vital signs are within normal limits.”

The NBME (2012) also provides the following example of authentic source materials in the form of a patient chart (p. 41).

Scenarios can be followed by a single question or a series of questions, such as in this example from Haladyna et al. (2002, p.321):

Imagine you are a delegate from Massachusetts to the Constitutional Convention. You have been authorized to act on behalf of your state.

  1. You would most likely approve of the
    1. New Jersey Plan.
    2. Virginia Plan.
  2. You would oppose the three-fifths compromise because
    1. Your state, as a rule, is strongly abolitionist.
    2. You will be grossly outrepresented in Congress by northern states.
    3. You want only a single representative house.
  3. You support the suggestion that Congress tax
    1. Imports.
    2. Exports.
  4. Because of your state’s experience with Shays’ Rebellion, you feel
    1. Farmers should not have to carry the tax burden for townspeople.
    2. Native Americans must be pacified before there can be peace.
    3. Tories ought to pay reparations.

Case-based or sequential question sets are those in which a scenario unfolds over time. As the scenario advances, additional information is provided for the student to consider.

Take a look at the following example from NBME (2024, p.43):

Initial scenario

A 35-year-old woman is brought to the emergency department because of worsening pain and swelling of the right knee for the past 2 days. She has been taking acetaminophen for the knee pain during the past 2 days, but the pain is worse today. She has not had any trauma to the knee or any previous problems with her joints. She is otherwise healthy, and she currently takes an oral contraceptive. She is sexually active and has a 10-year-old son who lives with her. She is 160 cm (5 ft 3 in) tall and weighs 52 kg (115 lb); BMI is 20 kg/m2. Temperature is 37.9°C (100.2°F). The right knee is erythematous, swollen, and tender, and there is pain with movement. No other joints are affected. X-ray of the knee shows an effusion but no structural abnormalities of the joint.

Item 1: Which of the following is the most appropriate next step in diagnosis?

  1. Arthrocentesis of the knee
  2. Blood cultures
  3. Complete blood count
  4. MRI of the knee
  5. Urine cultures
Evolving scenario 2

Arthrocentesis is done. The synovial fluid is cloudy. Gram stain is negative. Analysis of the synovial fluid shows a leukocyte count of 120,000/mm3 and 90% neutrophils.

Item 2: Which of the following is the most appropriate additional test on the synovial fluid?

  1. Culture for bacteria
  2. Glucose concentration
  3. Polarized light microscopy
  4. Protein concentration

Theme 3 takeaways

When writing MCQs to assess HOTS, a few strategies include:

  1. applying learning to new contexts
  2. combining information from multiple sources
  3. using authentic scenarios and/or materials
  4. creating sequential questions.

Conclusion

Multiple choice questions can be an appropriate method of assessing the higher order thinking skills of Applying and Analysing when used effectively. Before starting to write your MCQs, you should consider:

  • whether MCQs are an appropriate form of assessment in your context
  • which levels of cognitive abilities you are assessing through your ILOs
  • your overall formative and summative assessment approach.

When writing MCQs focused on assessing more advanced cognitive processes, a few strategies include:

  1. applying learning to new contexts
  2. combining information from multiple sources
  3. using authentic scenarios and/or materials
  4. creating sequential questions.

[1] Anderson, L. W., & Krathwohl, D. R. (Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.

[2] Biggs, J., & Tang, C. (2011). Teaching for quality learning at university (4th ed.). Open University Press.

[3] Bissell, A., Lemons, P.P. (2006). A new method for assessing critical thinking in the classroom. BioScience, 56, 66-72.

[4] Bloom, B.S. (1956). Taxonomy of Educational Objectives, Handbook: The Cognitive Domain. David McKay, New York.

[5] Crowe, A., Dirks, C., & Wenderoth, M.P. (2008). Biology in Bloom: implementing Bloom’s Taxonomy to enhance student learning in biology. CBE Life Sci Educ, 7, 366-381.

[6] Douglas, M., Wilson, J., & Ennis, S. (2012). Multiple-choice question tests: A convenient, flexible and effective learning tool? A case study. Innovations in Education and Teaching International, 49(2), 111–121. https://doi.org/10.1080/14703297.2012.677596

[7] Haataja, E., Tolvanen, A., Vilppu, H., Kallio, M., Peltonen, J., & Metsäpelto, R. (2023). Measuring higher-order cognitive skills with multiple choice questions –potentials and pitfalls of Finnish teacher education entrance. Teaching and Teacher Education, 122. https://doi.org/10.1016/j.tate.2022.103943

[8] Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5

[9] Haladyna, T.M. (2004). Developing and Validating Multiple-choice Test Items (3rd ed.). Routledge. https://doi.org/10.4324/9780203825945

[10] Lemons, P. P., & Lemons, J. D. (2013). Questions for assessing higher-order cognitive skills: it’s not just Bloom’s. CBE life sciences education, 12(1), 47–58. https://doi.org/10.1187/cbe.12-03-0024

[11] Liu, Q., Wald, N., Daskon, C., & Harland, T. (2024). Multiple-choice questions (MCQs) for higher-order cognition: Perspectives of university teachers. Innovations in Education and Teaching International, 61(4), 802–814. https://doi.org/10.1080/14703297.2023.2222715

[12] National Board of Medical Examiners. (2024). Item-Writing Guide: Constructing Written Test Questions for the Health Sciences. https://www.nbme.org/institutions/nbme-item-writing-guide/

[13] Sir Mar. (2023). BLOOM’S TAXONOMY. https://sirmarholanda.blogspot.com/2023/03/blooms-taxonomy.html

[14] Thompson, A. R., & O’Loughlin, V. D. (2015). The Blooming Anatomy Tool (BAT): A discipline-specific rubric for utilizing Bloom’s taxonomy in the design and evaluation of assessments in the anatomical sciences. Anatomical Sciences Education, 8(6), 493–501. https://doi.org/10.1002/ase.1507

[15] Zheng, A. Y., Lawhorn, J. K., Lumley, T., & Freeman, S. (2008). Application of Bloom’s taxonomy debunks the “MCAT myth”. Science, 319(5862), 414–415. https://doi.org/10.1126/science.1147852

Cookie preferences