When it comes to ensuring the efficacy of exams, one of the main priorities for test owners to consider is security. While state-of-the-art technologies—such as firewalls, system monitoring, etc.—and best-practice test center administration methodologies are taken to safeguard your content, certification and licensure organizations often look for test-delivery methods to further protect their exams.

One strategy these organizations often look to use to enhance test delivery security is response option randomization. This method of test security uses test delivery software to present the answer options to the same multiple choice question in a different order for each test taker. The underlying assumption in employing this strategy is that randomizing response options will mitigate the risks of test taker cheating, especially as a result of rote memorization of correct-answer options.  

While randomizing response options may be a method for increasing test security, there are academic studies that present reasons for why you may want to reconsider the approach as a test security measure.

Issues with Option Randomization

From our experience, there are two primary areas of concern with adopting option randomization to achieve an organization’s content security goals. The first has to do with item difficulty and test-form equating. It has been shown that an item’s difficulty level can be impacted by the position of the correct answer key, and that changing the order of presentation for answer options complicates the assessment and evaluation of the item’s true difficulty level. It is imperative that test development professionals be able to identify the difficulty of an item when creating test forms of comparable difficulty for test takers, as this can dramatically impact test taker scoring and in determining whether the individual passed or failed the examination.

For many testing programs, it is desirable or necessary to create multiple forms of an exam, and in those cases, it is important to ensure the forms are properly equated. To successfully equate test forms, it’s necessary to meticulously identify a block of “equator items” from older test forms to use on newer test forms. It is critical that these equator item blocks stay as consistent as possible on both the old and new forms—meaning no content alterations that change the items’ difficulties should be made to these equator items.

The other issue with option randomization has less to do with best-practice testing methodologies and more to do with human psychology. According to research, it’s found that when guessing, candidates have a higher tendency of selecting answer options located in the middle of the option set, as opposed to options on the edges. This supports the common adage among test takers, “when in doubt, choose option ‘C’. ” The findings from these studies suggests that the bias to choose middle items is large enough to have actual psychometric consequences, and potentially impact the fairness of exams that opt to include option randomization.

What Test Delivery Security Measures to Take Instead

Given these concerns with option randomization, testing organizations should consider applying other best practices that provide benefits to test security, while guarding against inadvertent impartiality and maintaining testing integrity. We recommend the following five options:

  1. Randomize item order- by changing the order of items on every test administered, you look to achieve the same effect as option randomization, without compromising the integrity of your exam results.
  2. Refresh content frequently- by altering your test content and developing new items on a consistent basis, you can help limit item exposure—thus increasing security and minimizing the risk of candidates already knowing the answers.
  3. Use multiple test forms that are equated- a low candidate-to-form ratio can deter test takers from stealing and sharing your content, since items are not guaranteed to appear consistently on your test forms. This is especially impactful for programs that deliver exams continuously—enabling the test owner to limit the amount of time any one form is in circulation. 
  4. Maintain minimal form-to-form content overlap- taking strategy three a step further, limiting how often you repeat content on your test forms better protects the overall integrity of your exam.
  5. Seek the guidance and support of a third-party assessment development provider- your organization invests significant resources, including time, money, and expertise, when creating your testing content; in many cases, it’s the primary intellectual property asset for your organization. You want to ensure that the end product provides the highest value for your organization and the constituents you serve. Engaging the support of an experienced, dedicated third-party test development professional may be a cost-effective way to support the development and execution of the other four strategies. This is especially true for organizations with limited resources and other strategic priories, such as membership services, professional education, and events.

 

Learn more today!

 

Authors
Keyu Chen, Ph.D.; Psychometrician, Assessment Services
Li-Ann Kuan, Ph.D.; Vice President, Assessment Services