This question really cuts to the core of our understanding of computer adaptive assessments such as STAR as well as the instructional use and interpretation of grade level standards.
- Computer Adaptive Tests (CAT): The assessment tool adopted by Imagine Schools (STAR by Renaissance Learning) is a CAT. This means that it is designed to adjust the level of difficulty of the assessment, based on the responses provided, to match the knowledge and ability of a test taker. The assessment begins at the difficulty point indicated by the most recent assessment or, if this is not available, the point represented by the student’s grade level. If a student gives a wrong answer, the computer follows up with an easier question; if the student answers correctly, the next question will be more difficult. This process continues until the difficulty level of the questions stabilize. This process is pictured in the following graph:
After about thirty questions, the computer has completed this difficult/easy search process and has been able to determine an estimate of the student’s true scale score. This scale score is based on the point at which the difficulty level stabilizes (right side of the chart) and has no relationship to how many questions the student answered correctly. Since computer adaptive testing systems select questions that are intended to be appropriately challenging for each student, most students will get about half the questions right and half wrong, so a score based on the total number or percentage of correct responses would be meaningless. Therefore, computer adaptive scoring is based on the difficulty of the items completed towards the end of the assessment. These assessments have been shown to be at least as accurate as the longer, fixed length tests (FLT) used previously.
It is important to understand that a CAT is a single assessment tool capable of adjusting to a student’s academic level regardless of their grade placement. A student in grade 5 may have their questions reduced in difficulty well below the grade 5 expectations or raised well above the grade 5 standards. This allows for the search process to establish an accurate scale score regardless of the student’s ability level related to grade placement. For this reason, the search process may take a particular student either well above or below the grade level expectations in his/her particular state and grade level in order to customize an assessment to meet their academic level. Unless the assessment presents items beyond a student’s capability, it cannot accurately zero in on the appropriate scale score.
This is much different than an FLT where there is a separate test for each grade level and the scale score determined is related to the number of items answered correctly. The FLT must be much longer in length in order to include questions that are both possible and challenging for a full range of students. Even with their highly extended length, FLTs have measurement issues for students who are very advanced or far behind and frequently result in either boredom or frustration for these students.
- Grade level standards: These standards should be interpreted as the minimum expectations for an average student. As such, they should not be interpreted as a ceiling on the growth of a student in a particular grade. For example, if Jose is in the fifth grade and is ready to move on to topics in the sixth or even seventh grade standards for his state, it is boring and unduly restrictive to prevent him from growing educationally as much as he can. On the other hand, if Johnny is not ready for the fifth grade standards, it is frustrating and detrimental to expect him to learn these standards if he has not mastered the prerequisite skills. This is the reason that the instructional recommendations from STAR are based on a student’s scale score rather than a particular set of standards that may be inappropriate for that student. Solid educational practice demands that we meet students where they are and help them to grow as far as possible.
Since the STAR assessment functions with the search process described above, it will continue on to present questions that can define the highest level of a student’s ability. For some students that have good skills in a particular subject, this should result is questions well beyond their grade level standards.