Why are my predictions frequently incorrect?
In an earlier blog post, we discussed the difficulties involved in making individual student predictions, especially when we are attempting to make pass/fail predictions.
Great care must be utilized when attempting to predict an individual student’s performance on a state test. Many idiosyncratic issues or occurrences can influence a student’s score on a single test and these sources of error combine when one assessment is used to predict another. When we are reducing the information in both assessments into a dichotomy (predict pass, predict fail to pass) these potential errors are magnified (from STAR Results and Proficiency Scores)
It is also true that::
teachers who work with ALL students using ALL of the information available (including STAR, focus assessments, checks for understanding, personal knowledge, etc.) and moving them as far as possible will be a winning strategy to optimize the results on any state test (from STAR Results and Proficiency Scores)
However, it is only natural to want to have an idea of whether or not Johnny will pass the state test. The typical means of reporting such predictions is predict pass (on track) or predict fail (not on track). However, this is too simplistic an approach and leads to frustration when the predictions appear to be incorrect.
Back when the weather service was putting out predictions of “it will (or will not) rain tomorrow” it was frequently the target of humor and developed a reputation for inaccuracy. When the weather service began to consistently provide forecasts in terms of probability of precipitation (POP), the true nature of mathematical prediction was presented and generally understood by the public. A prediction that there is a 30% chance of rain at noon tomorrow is vastly more helpful than collapsing that into a “no rain” forecast. Each person can make decisions based on a better understanding of the risks involved.
Our understanding of predicting a student’s success needs to evolve into a reflection of the true nature of prediction. Looking at prediction as a dichotomy (will pass, will not pass) completely ignores much of the information available as well as the sources of inaccuracy not included in the model (individual student circumstances). This view implies that there is such a thing as a sure thing. Even a top student can do poorly on one assessment due to a large number of potential factors. It should also seem obvious that a student with a high NCE score has a greater likelihood of passing a state test than a student with a low NCE score and that there is no magic line separating the two. By utilizing a probability to predict a student’s potential for passing a future test we focus our thinking on the fact that even a student who scores high on STAR can fail the state test and a student who scores low on STAR can pass the state test. The probabilities (not certainties) of these things happening are just relatively low.
We have the data to build a mathematical model that will result in a probability statement for a student passing the state test for our large states. A set of tables linking the NCE score to a probability of passing the state test in AZ, FL and OH are contained at the end of this blog. It is important to note that these probabilities are estimates based on the performance of Imagine students during the 2015-16 academic year. These relationships may change somewhat from year to year but the pattern of increasing probability of passing the state test as a student’s NCE score on STAR increases will not. For schools in the other states, observe that the patterns are very similar and vary according the proficient/not proficient cut point used in each state. States with a low proportion passing would have the probabilities lowered and states with a high proportion passing would have them raised.
Therefore, in Arizona, rather than saying that Julia, who scored a PR of 68 on STAR Reading is “on track” to pass the state test, it is more meaningful to report that Julia has a 65% probability of passing the state test (round to the closest value in the table). Similarly, if Frank scores an NCE of 44 on STAR Math, a helpful way of communicating his potential on the state test is to indicate that he has a 31% probability of passing. These probabilities should be interpreted as a student’s probability of passing barring significant individual events (e.g. student is sick or has a traumatic family event on the day of either assessment).
ARIZONA
| Reading/ELA | Mathematics | |||
| NCE | Probability of Proficient | NCE | Probability of Proficient | |
| 5 | <2% | 5 | <2% | |
| 10 | <2% | 10 | <2% | |
| 15 | <2% | 15 | <2% | |
| 20 | <2% | 20 | <2% | |
| 25 | 6% | 25 | <2% | |
| 30 | 14% | 30 | 7% | |
| 35 | 22% | 35 | 15% | |
| 40 | 30% | 40 | 23% | |
| 45 | 38% | 45 | 31% | |
| 50 | 46% | 50 | 39% | |
| 55 | 54% | 55 | 47% | |
| 60 | 62% | 60 | 55% | |
| 65 | 70% | 65 | 63% | |
| 70 | 78% | 70 | 71% | |
| 75 | 86% | 75 | 79% | |
| 80 | 94% | 80 | 87% | |
| 85 | 99% | 85 | 95% | |
| 90 | 99% | 90 | 99% | |
| 95 | 99% | 95 | 99% | |
FLORIDA
| Reading/ELA | Mathematics | |||
| NCE | Probability of Proficient | NCE | Probability of Proficient | |
| 5 | <2% | 5 | <2% | |
| 10 | <2% | 10 | <2% | |
| 15 | <2% | 15 | 3% | |
| 20 | 3% | 20 | 11% | |
| 25 | 11% | 25 | 18% | |
| 30 | 19% | 30 | 26% | |
| 35 | 27% | 35 | 34% | |
| 40 | 35% | 40 | 42% | |
| 45 | 43% | 45 | 49% | |
| 50 | 51% | 50 | 57% | |
| 55 | 59% | 55 | 65% | |
| 60 | 67% | 60 | 72% | |
| 65 | 75% | 65 | 80% | |
| 70 | 83% | 70 | 88% | |
| 75 | 91% | 75 | 95% | |
| 80 | 99% | 80 | 99% | |
| 85 | 99% | 85 | 99% | |
| 90 | 99% | 90 | 99% | |
| 95 | 99% | 95 | 99% | |
OHIO
| Reading/ELA | Mathematics | |||
| NCE | Probability of Proficient | NCE | Probability of Proficient | |
| 5 | <2% | 5 | <2% | |
| 10 | <2% | 10 | <2% | |
| 15 | <2% | 15 | 3% | |
| 20 | 7% | 20 | 10% | |
| 25 | 15% | 25 | 18% | |
| 30 | 22% | 30 | 25% | |
| 35 | 30% | 35 | 33% | |
| 40 | 37% | 40 | 40% | |
| 45 | 45% | 45 | 48% | |
| 50 | 52% | 50 | 55% | |
| 55 | 60% | 55 | 63% | |
| 60 | 67% | 60 | 70% | |
| 65 | 75% | 65 | 78% | |
| 70 | 82% | 70 | 85% | |
| 75 | 90% | 75 | 93% | |
| 80 | 97% | 80 | 99% | |
| 85 | 99% | 85 | 99% | |
| 90 | 99% | 90 | 99% | |
| 95 | 99% | 95 | 99% | |