Cut Score Rating of the A-CPT Exam

After training, the judges completed the 125-item examination. They were instructed to a) read each question, b) record their answer to the question, and c) rate each item as to...

"How many out of 100 borderline qualified test-takers will answer the question correctly?"

The response of the judge's results were tallied and presented to them.

A round of Angoff ratings were reviewed by the judges via web conference. A spreadsheet showing each panel member's obtained score on the examination and their first individual recommended cut score was presented to them. (Only the obtained score and cut score were shown; all identifying features of the panel members were removed and the sequence altered to protect the anonymity of the panel members).

The feedback report containing the panel ratings on each of the 125 items was displayed via the web conference for panel discussion of first ratings. For each item, the feedback report contained the answer keys and the frequency of ratings from the panel in categorical intervals (in percentage points) 0-20, 25-40, 45-60, 65-80, 85-100. The ratings for each item were examined in sequence and the panel was encouraged to discuss their differences in ratings, particularly when there was large variation in the panel judgments. During this discussion, the panel was asked to reexamine their original ratings in light of the discussion and data. They were to consider (1) the judgments of the entire group; (2) the definition of the minimally qualified candidate; (3) whether they answered the item correctly; and (4) the comments of the group. Then the judges were given an opportunity to change their rating, if desired, to reflect the group definition of the minimally qualified candidate. The process constituted the second round of ratings. The results of the second rating were submitted and tallied.