Advanced students (PR of 90 or above) can present challenges in regard to assessment and interpretation of the results. In the days before Computer Aided Testing (CAT), there were individual test booklets for each grade that were designed to be able to properly assess approximately 90 – 95% of the student population at that grade level. In order to properly assess an advanced student, a test booklet one or even two grade levels higher would have to be given to the student (this was called off level testing). This presented operational as well as interpretational challenges.
Now that were are using the STAR assessment system, off level testing is no longer necessary. All students, whether on grade level or not, can be assigned the same assessment and the computer will quickly make the adjustments to present an appropriate assessment. The scale score (the direct result of an assessment) will mean the same regardless of grade level. The resultant instructional recommendations derived from the scale score are not affected by grade placement.
The percentile rank (PR) and normal curve equivalent (NCE) are derived from the scale score and knowledge of the score distribution at each grade level. (Please see my “All These Numbers” blog post for a deeper discussion of these scales, their derivation, and meaning). While assessing a student in the upper 5% is no longer a problem, deriving meaningful standardized scores and measuring growth remains problematic. Since the PR scale tops out at 99 (and thus the NCE scale derived from it tops out at 99), students in the high 90’s will face this “ceiling” issue. It doesn’t matter how high their scale score is, the maximum NCE score will remain at 99.
The solution to this problem is “off level scoring”, where the standardized scores (PR and NCE) are determined as if the student were in a higher grade. This solution is pictured in Figure 1. Johnny scored a scale score of 620 and is in the third grade by age grouping. If standardized scores were to be derived for him as a third grader,(red rectangle), his PR and NCE scores would be close to 99 (the maximum) and he could show growth in scale score terms, but not in NCE’s (the ceiling effect). If Johnny were scored as a fourth grader (blue rectangle), his NCE score would be approximately 64 (corresponding to a PR of 75), leaving plenty of room to demonstrate growth in NCE terms and generate a learning gain greater than 1.0.
Care must be taken, however, in interpreting and explaining Johnny’s new scores. Johnny’s scale score and instructional recommendations will be accurate, as will his learning gains (due to the removal of the ceiling). Johnny’s PR and NCE scores will be lowered and should not be used for any decision regarding Johnny. In discussing Johnny’s scores with his parents, the reason for (too advanced to be scored on grade level) and results of (artificially lowering the standardized scores) off level scoring should be carefully explained.
Figure 1 – Off Level Scoring Example
Comments 4
Thank you for this post, it is very helpful in understanding off-level scoring. I am curious about the other reports we pull within Renaissance, does this impact the results when looking at grade level proficiency reports?
Off level scoring will impact the determination of norm-referenced scores (PR and NCE). The student will be treated in all reports as if he/she were actually in the adjusted grade level.
One problem we have found with off-level scoring is using it and trying to determine 4th quartile growth. Because a student who is normally close to 99% will drop to, possibly a 64%, when looking at 4th quartile growth, we aren’t really looking at growing our top kids. That 4th quartile student may have dropped to the 3rd, or even second quartile. We have to resort to Scaled Score, and looking individually to determine if what we are doing is truly growing those top students. In essence, off-level scoring changes who is in the 4th quartile.
You are correct. This will happen more often in the lower grades when the growth curve is steepest.
Remember, when using the scale score that students are expected to increase their scale score whether or not the student has been exposed to high quality educational experiences, just like children will tend to grow in height even if this growth is less than expected.
The only way to determine whether or not a student is experiencing above average educational experiences is to look at the change in the norm referenced score (PR and NCE). Off level scoring is intended to remove the ceiling effect so that this change can be observed. The appropriate measure of growth for a student scored off level is still the learning gain.