“Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests. Validity is, therefore, the most fundamental consideration in developing tests and evaluating tests.” (Standards for Educational and Psychological Testing, p. 11)
Since the pandemic began and the testing industry experienced a major shake-up, much discussion and research has centered around the validity of remotely proctored testing versus brick-and-mortar testing, but we’ve heard less about the validity of remote test development processes versus in-person test development processes. If you used to conduct most of your test development meetings in person but have shifted to virtual, have you taken a step back to consider the validity of your processes and whether or not you’re still getting the same result by performing them virtually?
In the world of test development, Standard Setting is one of the most important components. The Standard Setting process defines what a passing score is, and what it means that someone has passed the exam. Standard Setting connects directly to validity, which determines if we have the evidence to support using test scores to make decisions.
Among test development activities, Standard Setting is uniquely challenging to facilitate remotely due to the following considerations:
Security Concerns: Subject Matter Experts (SMEs) have full access to exam content.
Complexity: The process can be difficult to explain and digest.
High-level Engagement: Quality Standard Setting requires the highest level of SME engagement. Partial or passive participation will allow important procedures and considerations to be overlooked and unexamined.
Not Easily Revised: Once complete, the standard set for a test is very difficult and tedious to change. Established cut scores require a tremendous amount of effort to update, and once administered, any changes to cut scores may adversely affect candidates.
The case studies below illustrate three successful models for virtual Standard Setting, each involving unique challenges and solutions.
CASE STUDY #1: American Board for Certification in Orthotics, Prosthetics and Pedorthics (ABC)
- Special challenges for small-volume exam programs
CASE STUDY #2: Certification Commission for Healthcare Interpreters (CCHI)
- Special methods for the CHI™-Spanish exam
CASE STUDY #3: National-Interstate Council of State Boards of Cosmetology (NIC)
- Specific processes and challenges of a semi-remote standard setting for practical examinations
For context, there are 8 key steps in a typical Modified Angoff Standard Setting process:
- SME recruitment
- Training
- Discussion of the minimally qualified or borderline candidate
- Practice ratings
- Angoff ratings: round 1
- Group discussion of flagged items
- Angoff ratings: round 2
- Final recommended cut score
CASE STUDY #1: American Board for Certification in Orthotics, Prosthetics and Pedorthics (ABC)
ABC has multiple small-volume exam programs in orthotics, prosthetics, and related disciplines. Certificants may work technical jobs, have varying levels of education, and some even work in retail-type settings.
Remote Standard Setting provided a number of advantages for ABC’s programs:
- SMEs don’t need to travel
- Saves on costs and coordination effort
- Ability to tap a much broader constituency of SMEs
- Ability to replace SMEs with little notice
Remote Standard Setting also presented ABC with unique challenges:
- Sessions need to be broken up into multiple days/times
- Maintaining engagement of SMEs is more challenging
- Technology needed to participate may present challenges
- SMEs who work in retail or clinical settings may be unable to commit to calls during work hours
- SMEs may not have access to private space to be on calls
ABC recommends addressing the above challenges to engagement by being more proactive in planning ahead, creating a schedule that’s tailored to the SMEs and may be outside normal business hours, and maintaining engagement throughout the process. Consider having a “technology pregame” meeting to test connections and troubleshoot to build confidence in meeting and collaborating virtually. Clearly explain the required commitment in advance to potential SMEs, including the need for them to be in a private space.
By design, in-person standard setting meetings allow for focused attention by SMEs, so the most challenging difference remote standard setting meetings present is facilitating the same level of engagement and communication experienced in effective, in-person standard setting meetings.
CASE STUDY #2: Certification Commission for Healthcare Interpreters (CCHI)
CCHI’s Spanish exam requires special methods to address exam peculiarities such as the bilingual (Spanish-English) performance component, audio recording format, and the fact that it is human-rated. These conditions created challenges for CCHI in the remote standard setting process, specifically in SME recruitment and training.
Unique Challenges in SME Recruitment and Training:
- Security concerns require a special emphasis on SMEs’ personal & professional integrity
- SMEs’ literacy in online platforms and interacting with audio
- Required significant speed of broadband connection
- Delivering an audio exam online to SMEs (extra test configuration; IT issues)
- Group interactions are somewhat weakened (compared to in-person) as there is no access to other SMEs outside scheduled meetings
Bilingual exams and exams that require similar format require a nonstandard process for both remote and in-person modalities. In the remote modality, an orientation meeting is needed to set clear expectations and address process and technology aspects of the project. There will more than likely be a need for additional training about the human rating process, scales and rating conventions. In this case, the Extended Modified Angoff method supplemented by the Beuk Relative-Absolute Compromise method was the most effective approach to determine cut scores.
CASE STUDY #3: National-Interstate Council of State Boards of Cosmetology (NIC)
The remote standard setting protocol used for the NIC Practical Examination is an example of the specific processes and challenges of a semi-remote standard setting for practical examinations. The NIC Practical Examination is an in-person performance exam used for Cosmetology and related fields. It is used in multiple states across the U.S., and typically administered in board offices or hotel conference/ballrooms. The exam is comprised of multiple timed sections, and candidates are responsible for bringing their own supplies. A proctor and raters are required, and a mock-candidate run-through is included in the standard setting process.
The standard setting process for these exams changed to a hybrid in-person/remote model due to the pandemic and was accompanied by a number of new challenges. Prior to the pandemic, all test development activities had been performed in-person and practical standard setting was the only test development activity determined by NIC to require an in-person presence for the mock-candidate run-through. There was some question as to whether conducting the exam run-through over webinar in real time or a pre-recorded run-through would be as effective as the SMEs seeing it in person. Ultimately, NIC opted to find a new location for SMEs to meet and have a facilitator join via webinar.
The hybrid remote/in-person standard setting process included a preliminary SME training, as well as review of the critical standards and target candidate. A room was set up to mimic an examination setting and roles (such as proctor, examiners, and timekeeper) were assigned for the mock-candidate run-through. The mock-candidate run-through was performed and observed, rated, and then discussed by SMEs. A Q&A session with the mock-candidate followed. Angoff ratings were provided by SMEs and the mock-candidate before ending with a group discussion and adjustments, and then a final cut score was established. Even with the virtual adjustment of the required test development activities, there were several challenges to the remote standard setting process.
Challenges to Remote Standard Setting
- Limited access to facilities in advance
- Technology and/or Audio Visual equipment may be unfamiliar
- Printing updated materials is more time-consuming and possibly costly
- Facilitator ability to provide knowledgeable input is restricted
- Barriers to SME engagement
- Secure document distribution and control
- Some SMEs less likely to attend in-person at this time
- Additional responsibilities fall to Client Representative
Virtual standard setting can be a program-saving tool when executed properly, with careful consideration of the differences between in-person and virtual processes.
Key Considerations in the Remote Standard Setting Process:
- SME recruitment: Thoroughly screen the SMEs to ensure you can trust them to access test content from home, and recruit more SMEs than needed in case of last-minute attrition.
- Communication and SME engagement management before the event: Clearly outline the level of commitment so that SMEs aren’t surprised by how much time and effort is involved in a Standard Setting.
- SME engagement during the event: Consider how you can optimize participation at all points in the process, such as requiring or strongly encouraging camera use to mirror in-person engagement.
- Technology: Clearly outline the requirements ahead of time, and consider a “technology pre-game” meeting to ensure everyone’s full participation. Ensure that facilitator-driven technology use and transitions are seamless so that no time is wasted.
- Security: Consider that there is an increased level of risk inherent in remote Standard Setting. Ensure that SMEs sign NDAs and understand their responsibility, and screen them thoroughly during the recruitment process.
- Creative solutions for unique situations/exams: Every exam is unique, but it’s possible to adapt the virtual standard setting modality to every situation – even if a hybrid model is needed for a practical exam!
And ultimately, the most important consideration:
Validity: Will you get the same level of standard setting process validity in a virtual model? Will you get the same product (i.e., the same cut score) as you would if you conducted the standard setting in person?