This question really cuts to the core of our understanding of computer adaptive assessments such as STAR as well as the instructional use and interpretation of grade level standards.
- Computer Adaptive Tests (CAT): The assessment tool adopted by Imagine Schools (STAR by Renaissance Learning) is a CAT. This means that it is designed to adjust the level of difficulty of the assessment, based on the responses provided, to match the knowledge and ability of a test taker. The assessment begins at the difficulty point indicated by the most recent assessment or, if this is not available, the point represented by the student’s grade level. If a student gives a wrong answer, the computer follows up with an easier question; if the student answers correctly, the next question will be more difficult. This process continues until the difficulty level of the questions stabilize. This process is pictured in the following graph:
After about thirty questions, the computer has completed this difficult/easy search process and has been able to determine an estimate of the student’s true scale score. This scale score is based on the point at which the difficulty level stabilizes (right side of the chart) and has no relationship to how many questions the student answered correctly. Since computer adaptive testing systems select questions that are intended to be appropriately challenging for each student, most students will get about half the questions right and half wrong, so a score based on the total number or percentage of correct responses would be meaningless. Therefore, computer adaptive scoring is based on the difficulty of the items completed towards the end of the assessment. These assessments have been shown to be at least as accurate as the longer, fixed length tests (FLT) used previously.
It is important to understand that a CAT is a single assessment tool capable of adjusting to a student’s academic level regardless of their grade placement. A student in grade 5 may have their questions reduced in difficulty well below the grade 5 expectations or raised well above the grade 5 standards. This allows for the search process to establish an accurate scale score regardless of the student’s ability level related to grade placement. For this reason, the search process may take a particular student either well above or below the grade level expectations in his/her particular state and grade level in order to customize an assessment to meet their academic level. Unless the assessment presents items beyond a student’s capability, it cannot accurately zero in on the appropriate scale score.
This is much different than an FLT where there is a separate test for each grade level and the scale score determined is related to the number of items answered correctly. The FLT must be much longer in length in order to include questions that are both possible and challenging for a full range of students. Even with their highly extended length, FLTs have measurement issues for students who are very advanced or far behind and frequently result in either boredom or frustration for these students.
- Grade level standards: These standards should be interpreted as the minimum expectations for an average student. As such, they should not be interpreted as a ceiling on the growth of a student in a particular grade. For example, if Jose is in the fifth grade and is ready to move on to topics in the sixth or even seventh grade standards for his state, it is boring and unduly restrictive to prevent him from growing educationally as much as he can. On the other hand, if Johnny is not ready for the fifth grade standards, it is frustrating and detrimental to expect him to learn these standards if he has not mastered the prerequisite skills. This is the reason that the instructional recommendations from STAR are based on a student’s scale score rather than a particular set of standards that may be inappropriate for that student. Solid educational practice demands that we meet students where they are and help them to grow as far as possible.
Since the STAR assessment functions with the search process described above, it will continue on to present questions that can define the highest level of a student’s ability. For some students that have good skills in a particular subject, this should result is questions well beyond their grade level standards.
Comments 11
Thank you for this article. Sadly, we still have far too many teachers who do not understand these types of tests and their ability to be used as a useful diagnostic tool.
Thank you. You have identified a challenge that we all have to provide the professional development opportunities so that all of us can learn to use these tools as effectively as possible.
This explains a lot. Now I understand how these scores work.
Excellent! Thanks for the feedback.
First I would like to say thank you for all the work you put into designing a program for assessing and monitoring the learning of all the students in Imagine. I understand better how STAR works, my greatest concerns are when we have students on IEP’s that have a hard time reading and use accommodations for the test. It does not give a true picture of where they are in the learning because many kids can answer the questions when we are read the test questions and answer options. Some of the kids who can not read at all can figure this out and get to green( I had one who did) This was not a true picture of where he is in his learning and it was not data I could use to write true goals for him. He was much lower than it portrayed him to be. I almost would prefer to have a scaled score without accommodations to see a true picture of where they are independently. We would then have some students who would not be able to even pass the pretest to even take the STAR.
Would these students be better assessed on the Early Literacy program so we can help them in areas they need since they are truly non readers?
Thanks. You have identified one of the real challenges in designing IEP’s that allow for assessment results that can optimally lead to educational growth.
Thanks for this! Teachers are often frustrated by this feature of CAT. Sometimes they feel that the students should only be assessed on what they have already taught—not the content that may be appropriate for their level. I will definitively share and discuss this with my coaches.
Thank you. I hope that this can be of assistance in generating understanding and to move teachers toward an understanding that it is important to meet students where they are and take them as far as possible.
Thank you for sharing this explanation and visual! This is a great resource to refer for our teachers and assists them when speaking to parents with concerns.
Thank you! Little by little we will get everyone on board!
Stephania,
I love the idea of teachers using this as a visual when explaining the assessment and gains to parents. Spread the word!