Oklahoma takes a step back for education’s future

by Wendy S. Pratt

The July 2024 change in cut scores for state tests of English and math involves complicated statistical analyses into which few would delve. For Oklahoma Secretary of Education Nellie Tayloe Sanders, however, it is personal.

At the May 21 meeting of the state’s Commission on Educational Quality and Accountability, Sanders shared a message from an older friend. The friend, knowing Sanders has dyslexia, urged her saying she is particularly suited to help all Oklahoma children who struggle with learning.

"Oklahoma has struggled with reading for a hundred years,” yet it is "the access children have to learning," Sanders said. Sanders is a parent of two children who also struggle with reading, at a time when the Commission she leads is grappling with a process "complicated by politicians, politics, attorneys and bureaucracy."

Sanders encouraged members to "ask the hard questions and tell the truth about what has happened" when performance levels on the Oklahoma School Testing Program (OSTP) assessments were set for 2024. “Since we haven't gotten it right in 100 years,” she said, this is an opportunity for the CEQA to “align outcomes to expectations” so parents and other stakeholders have sound data upon which to make decisions.

Chad Buckendahl, Ph.D., who has been designing, evaluating, and working with educational testing programs for two decades, presented the findings for ACS Ventures, LLC, a national leader in assessments and credentialing. The ACS team worked with WestEd (a widely respected nonprofit research, development, and service agency of nearly 60 years). The team was contracted by the CEQA to determine the cut scores for each subject and grade level per Title 70 OK Stat § 1210.541 though authority for the process lies with the OSDE.

Dr. Buckendahl told Commissioners the Oklahoma State Department of Education (OSDE) "hasn't substantively cooperated" with the study nor provided evidence requested. Researchers were were denied access to interview members of the Oklahoma Technical Advisory Committee, a federally required team of experts that regularly determines if the state tests are “valid, reliable and fair.”

Two big problems were defined clearly by evaluators:

• The OSDE did not involve CEQA in the Design and Execution phases of the standard setting process as required
  in the legislation.

• Evidence from the state’s technical report and stakeholder interviews indicate a policy change from 2023 to 2024
  resulted in widening the “Honesty Gap.”

The dotted lines at the bottom of the ACS team’s graph (above) are Oklahoma’s National Assessment of Educational Progress (NAEP) scores for English Language Arts at Grade 4; the solid lines are OSTP scores for the same grade levels and years assessed. The gap between Oklahoma’s national and state performance is now greater than it was in 2017, which is clearer in their bar graph.

A less dramatic but clear widening of the achievement gap between state and national assessments was shown for 8th grade ELA as well as 4th and 8th grade mathematics.

Reasons cited in support of returning to the 2023 cut scores (also called “cut points” by the OSDE) were:

• State test blueprints were generally stable

• PLDs were clear

• Formula was aligned to NAEP benchmarks prior to 2024

• Returning to 2023 cut scores “maintains historical standard and reduces influence of pandemic effect.”

These are key factors if you want to compare a school’s proficiency results over time - for analysis of program effectiveness - and/or to support parental-choice decisions. [For more, see previous article here.]

Testimony from Dr. Ashlyn Fiegener, commissioner from Tulsa who represents higher education, was particularly revealing. In mid-June 2024, she attended the state’s Standard Setting meeting in Broken Arrow. The full standard-setting process is defined and implemented by the OSDE with the current state-test vendor, in 2024, that was Cognia. Fiegener stated, "We did not have enough information to go on to say they [the cut scores] were inappropriate.” This contributed to why the CEQA did not reject the cut scores recommended by the OSDE last July.

According to the OSDE’s 2024 Oklahoma Standard Setting Report: OSTP ELA and Mathematics — Grades 3–8, "Based on the recommendations of the Oklahoma Technical Advisory Committee, Cognia psychometricians calculated and then presented adjustment options to OSDE for their consideration. After discussion and review, the OSDE made no policy adjustments to the articulated standard setting results." This is just one indication of the substantive disagreement among experts about where Oklahoma’s cut points for “Proficiency” should be set. The controversy has not gone unnoticed nationally. In “Skyrocketing Test Gains in Oklahoma Are Largely Fiction, Experts Say,” non-profit news outlet The 74 detailed how Cognia was making plans to work with OSDE staff on explaining the change but apparently that didn’t happen.

Sanders asked Dr. Buckendahl how much information he would estimate the CEQA members had received from the OSDE as a percentage of what they would need to know to make such a decision. The ACS assessment evaluator said it was likely "less than half.” Shortly after, Secretary Sanders made the motion to "reject the 2024 cut scores and reinstate the 2023 cut scores since they align to NAEP, close the 'honesty gap', and ensure that students achieving Proficiency are on track for College and Career Readiness." The motion was unanimously approved.

Bottom line: What do psychometric standard settings have to do with helping students who cannot learn to read or read to learn? Everything. Only with accountability data that is valid, reliable and fair can appropriate decisions be made regarding academic needs or goals for students, schools or states.

The CEQA’s action will keep state-test scores comparable from year to year and provide a closer alignment to the NAEP, which is only important to preparing students for college and careers OR if a state would like their national rankings to improve. Because the NAEP is given to a small sample of students in each state (2,500 in Oklahoma), the ONLY way to improve in national rankings based on it is for ALL students to perform better on state tests. So, this is one of those times when taking a step backward may ensure public education in Oklahoma can move forward.