Many Standardized Tests do not Measure What they Should

Reason 2/5: Why Making the Atlanta Public School Teachers Felons Won’t Make a Difference:

Ever looked at a high school’s English curriculum? The objective strands look like a list of “shoulds”: Child will be able to synthesize primary and secondary documents. Child will be able to recognize post-war poetry elements. Child will be able to use punctuation properly. Child will be able to properly conjugate helping verbs. There are scores of these objective strands.  And on a standardized post-test, because there are so many strands, and they all need to be represented, each one of those elements gets usually just one question. So apparently using proper verbs is equal to recognizing Post-War poetry elements; worse, if the child misses the single question about verbs, then apparently he knows nothing about verbs, and I’ve taught him nothing about verbs.

A strand that says the child can develop a thesis and defend it with strong writing cannot be accurately tested on a multiple choice exam. So wrongly, in order to assess that strand, the child is asked to find the thesis statement in an offered essay. Test writers often do not spend enough time writing these questions well, and honestly, there really might be two or three sentences that could work as a thesis statement. As any good writer knows, essays don’t really have just one sentence that guides the whole flipping essay. (Tell that to the testing boards.) So, based on these standardized tests, a kid who can select a thesis out of a choice of four answer options, but not ever write an essay is deemed just as educated as the kid who writes well.

One year a major writing test that our county system gives to tenth graders had a major flaw. Over forty-five percent of the kids went down the rabbit hole created by this flaw and failed. Instead of retesting the whole population with a valid test, because there were already scheduled retests in place for kids who fail, the board did nothing about this flaw; they let the failures go on the children’s records and simply retested those students. The next year a new question was developed and applied. 85 percent of the kids passed on the first attempt. Suddenly, our county was bragging about how much better we were all doing as educators. LOOK how high this year’s scores trump last year’s! Aren’t we grand?   These scores didn’t represent what they are supposed to, at all.

So not only are objectives unbalanced in terms of importance, not only is success defined inaccurately, but we misuse the data collected.

How does the APS judge expect teachers and children to take any of these tests seriously when we know we are all being misrepresented?

Next up: Reason 3/5 Why Making the Atlanta Public School Teachers Felons Won’t Make a Difference:

The Tests Dumb down the System in General

Why Making the Atlanta Public School Teachers Felons Won’t Make a Difference

Good teachers don’t worry about being measured and evaluated. But I have to clarify, we don’t worry about valid, worthy testing, testing that actually measures what it says it will, and measures knowledge that is relevant. Right now those sorts of measurements don’t exist

The teachers from the Atlanta Public Schools system deserve what is coming their way. . .To a degree. Did they know at the time they were committing something immoral and unethical? Did they know they could lose their jobs and careers? Yes. They had to. Did they know they were committing a racketeering felony? Probably not.

Elementary school teachers are not famous for being overly educated. Yes, I know plenty who are sharp and brilliant. But too many of them graduate from local paper mills with low entry and exit standards,; they earn what many college students know is the least intellectually challenging degree offered (if not the most tedious):a Bachelor of Education. (Face it: a course called The Theory of Education is never going to be as challenging as Thermodynamic Chemistry, nor Math for Children as deep as Legal Statistics.) So more than likely, those fraudulent APS teachers didn’t think clearly enough to wonder if they could be arrested.

What law did they think they were breaking? None. I’d bet they believed fraud laws were all about money and theft. Silly them. But then when they were arrested, they didn’t believe they’d go to jail, be convicted. 21 of 36 defendants admitted they were guilty before trial. Of the eleven remaining, only one of the accused took the D.A.’s bait to plea bargain down his sentence. The other defendants took their chances and went to trial. AND then, after being convicted and called Felons, when offered a chance to bargain again for their punishments, no one took that offer either. They believed perhaps they could go win an appeal? On what grounds?

I understand that sometimes when one feels righteous, one doesn’t want to knuckle under. I also get that sometimes the plea bargainer affects the outcome of those who choose to go to trial. But ultimately, these folks did commit fraud. They did racketeer. And they did so for financial gain (their incomes.)

And my bet is that they did this with impunity because this sort of illegal manipulation is overly common. It’s the “everyone is doing it” defense.

Here is my beef:

The judge thinks he is helping the children of our future with his severe reaction of long jail times. Perhaps. But the system of testing, and performance-for-pay is so broken, has so many flaws and loop holes, that his ruling barely makes a ripple in improving the structure. Yes, it might stop a group of teachers and administration from sitting around a table and openly defying the rules in the future. But that doesn’t mean the testing currently in place is now going to help the children at all anyway.

Five Serious Reasons why:

  1. There are still plenty of ways to cheat.
  2. Many Tests do not Measure What they Should
  3. The tests dumb down the system, in general.
  4. The children rarely fail anyway nor get remediation even if they fail the test.
  5. The tests can be manipulated to fire or secure workers who don’t deserve it (see number 1)

Teachers know this, so it is hard for them to feel supportive, patriotic, even ethical when it comes to testing. It’s like being asked to take the gossip on FB or Nightline as serious.

Let me break this down in a series of Four separate posts, if you care. This knowledge is important for anyone who can affect the political machine called education.

Reason 1/5: There are still plenty of ways to cheat.

These Atlanta folks got caught because an AJC reporter who was paying attention began to wonder what this one elementary school, a school that had for several years in a row gained in their scores by leaps and bounds over its comparable sister schools; if the gains were real, why wasn’t APS requiring the sister schools to do whatever it was that the “successful school” was doing, the reporter asked. But there was no difference in anything they did. And from there the story began.

Someone at these sister schools blabbed. See, teachers shift schools, though they are still employees of the same system. They develop allegiances and conflicts with former bosses. Someone who used to be in the law-breaking circle ratted them out. But even if the tattler hadn’t come clean, there is an independent group that can take the scantrons from the tests and do a comparative match. Too many erasures in the same places? Too many kids getting number 36 right, when number 36 is predicted to be the question that only half the kids will know? Hmmm.

So cheat method one is to literally change the answers. APS did this as a group. But there can be one principal who does this him/herself in the wee hours of the night. Who would know if no one is auditing the scantrons?

When I have had to proctor high stakes tests, here is how it has often worked. I go collect my testing materials. I sign them out. I get a box with a particular number of scantrons, pencils, test directions, test booklets. I sign something that says I won’t even look at the test. I administer the test. And at the end of the testing period, sometimes at the end of the day, I return all materials.

BUT imagine Student Bobby is the first to realize there is an odd typo in test question 32. Then Suzy. I’m not supposed to look at it. Do I go ahead and try and help them decipher the question? George shouts out the answer anyway. I have to report this “aberration in the testing environment.” Glancing at the questions to see if it is ethical to help overcome the typo, I realize George is wrong. He might be shouting out the wrong answer on purpose because he is a brat. Who knows? Do I help? I don’t know. But either way, I could easily tell everyone the answers.

I’m supposed to proctor everyone else’s students, not my own, in order to “maintain the integrity of the testing environment.” In otherwords no favorites. But I’ve been around the school and community for a long time. The kids all know me. I’ve taught some before. I taught their siblings. I hate this test. I go ahead and answer their question when they get stuck and try to get them to “figure it out” with hints. They are so stressed out. . .am I less or more ethical than the convicts from APS? This method of cheating could show up on an audit, but probably won’t because it is one class set of tests.

I return my boxed set to the administrative helpers (teachers who are not supposed to be teaching any of the test takers currently, so they won’t be tempted to cheat.) They pass it along to the testing administrative secretary, who supposedly locks it up for lunch until the testing administrator comes back, and they count all the booklets triple times, sign a bunch of documents, and lock them up for the night; someone from the county delivers them to a central location later when absentee kids get a chance to take their test.. At this point, three separate people besides me at the least have touched my tests, and each has had the chance to change answers without me knowing. (Did this happen to any of the APS crew?) An audit might be able to catch this one too, though it cannot tell anyone who was responsible.

Or kids themselves, knowing the proctor doesn’t know who they are can send in another kid to take their tests. One kid takes the math tests, the other the LA tests. We don’t check IDs and you’d be surprised how few students will rat on each other.

Plus, every now and then, we get to proctor our own kids, especially if the tests are given online. These labs get used over and over for tests through the day, and kids will literally write the answers on the desks for the next incoming group. Kids have pens with cameras in them now, too, and can sometimes pass along whole test pages. I’ve heard of teachers who will write certain answers on the board without drawing attention just to see if the kids will notice. And some do. “Hey, look. . .24. A.” Of course, these accusations can never be proven after the fact except through an audit, or a confession, which a school system never asks for.

But here’s another way many teachers can cheat. At my school, teachers are assessed by the jump from a pretest score to a post test score of a set of kids. The system decides what this group of kids should be able to score once they have learned. The closer I can get them to that score, the better. BUT it is even more rewarding for me as a teacher to have a huge jump. So if my illiterate kids score a 12 on the pretest and a 70 on the post test, this is better for me than my gifted kids who score a 91 on the pretest and then a 98 on the post test. And guess what, since the pretest is not “secure” I get to give it to my own students.

I’ve known of teachers who literally tell their kids to “Christmas tree” the pretest. It takes two days to give. I’ve also known of teachers who give it one day, and if the kids don’t finish, oh well. SO those cheating teachers look like they are masters when the post test comes around, because obviously, the kids will do better on the post test which gets averaged into their grades than the pretest they were told to Christmas tree. And guess what? Noone wants to dig into my accusations of this unethical behavior, though auditing pretest scores , might easily show what I am describing.. I cannot prove it after the fact myself if the accused won’t confess.

Again, the “everyone is doing it” defense makes it hard for teachers to put any serious belief behind testing. The judge’s harsh punishment of these teachers, though warranted, as affected this beliefminimally.

Next up: ”Reason 2/5 Why Making the Atlanta Public School Teachers Felons Won’t Make a Difference: Many Tests do not Measure What they Should.’”