“A BAD NUMBER IS WORSE THAN NO NUMBER AT ALL”
A short story:
On a trip to London, I throw caution and budget to the wind and head into the shop of one of the finest tailors on Seville Row to have a new suit made. After the initial pleasantries, the tailor asks me what my color and style preferences are and the primary purpose for the suit. Based on his understanding of my responses, he pulls out bolt after bolt of fabric for my inspection. After a discussion, I choose my desired fabric (quite beautiful, I might add). The tailor and his assistant then proceed to take extensive measurements and record them on a standard form. The accuracy of these measurements is vital to the proper fit of the suit and he and his assistant take care to always make them in the same manner. He notices that there is a 1/2” difference in the lengths of my arms, a fact that explains why one of the cuffs of my off-the-rack suits always seems a bit off. When I come back for a fitting a week later, the tacked together suit structure (without arms) fits like a glove and needs little alteration. After another week, I am able to try on the final product, which, I must admit, looks awesome on me.
This entire process and the success of the finished product is dependent on the consistency and accuracy of the two measurements (the initial interview process is a type of measurement itself). There will, of course, always be some inaccuracies (perhaps + 1/32”) but with a skilled tailor, these will be completely unnoticeable. This is why some on-line tailors using a phone app to measure claim to be more accurate. A good tailor will always re-measure each time I come back for a new suit as body dimensions tend to change over time.
Let’s move the story along and assume that something unusual happens. The assistant writes down 21 3/4” for my right arm length rather than the correct 25 3/4”. If the tailor notices it (and it should be obvious) he will immediately remove the bad number from his form and call me in to remeasure. If I am not available, he will still have removed the bad number and will perhaps use my left arm length, realizing that some alteration may be necessary at the final fitting. He knows that leaving a bad measurement on his form will cause far more serious problems than removing it will.
Why this story?
Good teachers are very much like tailors. They tailor a custom curricular approach for each of their students and classes. This customization must be based on consistent and accurate measurements of each student’s position on the educational journey. The consistency (standard administration procedures) and recency of the measurement are as imperative to the educational tailoring process as it is to the suit tailoring process. So too is the necessity of erasing (we call it deactivation) bad data. Leaving bad data on a student’s record is far worse than removing it, even if re-testing is not a viable option. Not only will a bad assessment result lead to inappropriate curricular tailoring decisions, it will result in seriously misjudging a student’s growth trajectory and amount of growth from that point on.
How do we know that an assessment result is bad?
The first line of defense, so to speak, is the classroom teacher. Quickly reviewing the results of an assessment period should reveal that a PR of 1 after spending 300 seconds on the assessment is inappropriate for Jose or an NCE of 25 is obviously inappropriate for Julie, who always scores advanced or above. In order to allow school leaders to assist in this bad data identification process, a Potential Outliers workbook is available in Tableau that will identify:
- Test scores paired with testing times that are very low, potentially indicating that the student “blew off” the test (Jose perhaps? ). It will also identify
- Students whose current score is more than 30 points away from their previous mean (Julie?).
If these scores are identified as truly unrepresentative of a student’s knowledge and skills, they should be deactivated. If at all possible, the student should be re-tested.
Remember:
“A BAD NUMBER IS WORSE THAN NO NUMBER AT ALL”