Background

Region 10 Education Service Center (ESC) in Texas plays a pivotal role in fostering student success and shaping the educational landscape across the state. As a partner to twenty other ESCs, Region 10 serves 120 public and charter school districts, over 3,000 schools, and more than 900,000 students across the state of Texas. The ESC creates, distributes, and supports an item bank, TEKSbank, that is used by 600 school districts and charters, over 5,000 schools, and more than 3 million students. From curriculum development and instructional support to leadership training and special education services, Region 10 is committed to empowering educators, enriching learning experiences, and driving positive outcomes for students of all backgrounds and abilities.

The Challenge

Region 10 publishes an item bank, TEKSbank, for school districts to help prepare students for the high-stakes exam, State of Texas Assessments of Academic Readiness (STAAR®). The 90,000 TEKSbank test questions are aligned to the state’s curriculum standards, Texas Essential Knowledge and Skills, as well as other tags like Webb’s Depth of Knowledge (DOK) and Bloom’s Taxonomy.

As educational trends change and the state updates its standards, Region 10 puts a lot of effort into making sure the bank is aligned and relevant.

To ensure content aligns with the state’s curriculum standards and contains various levels of difficulty, Region 10 conducts multiple alignment projects per year. Contractors are hired to analyze each item manually. Some projects require a single curriculum expert for each of the 12-grade levels to review as many as 2,000 items. Not only are these review processes time- consuming, but the manual review process led to a lack of accuracy and consistency among the updated items.

Region 10 was spending a significant amount of time and resources on maintaining content alignment, with little time to prioritize content growth and improvement. It was not uncommon to spend $150,000 annually just to update standards or tags on existing content. This became even more important in 2023 when the educational landscape of Texas underwent a significant transformation with the introduction of technology-enhanced items (TEIs) and online assessment. This was a departure from traditional assessment methods and challenged educators to prepare students for a new paradigm that emphasized critical thinking and a sophisticated demonstration of knowledge. To support teachers and lead them into this new frontier, Region 10 was faced with the challenge of creating 20,000 high-quality TEIs.

Region 10 initially outlined four primary goals that they were trying to achieve by sourcing a new partner.

• Ensure alignment to the latest state standards
• Identify knowledge gaps
• Assign depth of knowledge (DOK) Update content with cultural relevance

The ESC also needed to consider several important factors before they began the implementation of new technology to streamline practices:

• A need to maintain human oversight throughout the process
• Constant adjusting and perfecting of the content
• Reduce the time of alignment projects to allow more time for enhancing and developing more questions

Region 10 connected with Prometric’s Finetune team in March 2023. Finetune quickly understood the ESC’s challenges and began work to implement Finetune Catalog™, an AI-human hybrid tool for content alignment with in-depth insights across item banks and learning materials.

"Finetune had a really good understanding of K-12 content and the standards. It was easy to have conversations with them about what we wanted even though we didn’t have the technical AI knowledge of how to describe it.” 

- Jim Newhouse, Former Region 10 Assistant Director of Data Solutions

Outcome

Region 10 provided Finetune with learning materials such as exam blueprints, standards, item banks, and more. Finetune took this content and built a customized Catalog AI model that would work to analyze their content and see how well it aligned with Webb’s Depth of Knowledge. After the first alignment, Catalog reported a 96% accuracy rate, leaving Region 10’s team excited and optimistic for the future development and improvement of their content. Those “needles in the haystack” that needed alignment improvement were fixable, and they would be able to do it quicker, cheaper, and more precisely than ever before.

Along with the accuracy of the first alignment, the ESC was provided a rationale that would be useful in the building of future assessments. The complex rationale has become a transformative new collection of data that they are able to supply to test builders and teachers through the assessment platform.

“The implementation of Catalog took only a few weeks instead of the projected six months and reduced the number of contractors needed from ten to only three. Using Finetune Catalog to accomplish alignment tasks has freed up time and money and increased overall productivity.”

- Jim Newhouse, Former Region 10 Assistant Director of Data Solutions

Now, with savings of around $100,000 per core content area, including math, English language arts and reading (ELAR), science, and social studies, Region 10 can reinvest in producing new content to grow its current item bank. TEKSbank is now equipped with 25% (20,000) cutting-edge, tech- enhanced questions, which provides deeper knowledge assessment and a more accurate evaluation of student skills.

“We think of AI as a tool that is programmed by humans, overseen by humans, and analyzed by humans.”

- Jim Newhouse, Former Region 10 Assistant Director of Data Solutions