The Secrets of the CSAP

This week security is has been tightened at certain public buildings in Fort Collins as highly sensitive material is delivered under constant guard and stored under lock and key. These facilities are not airport or government buildings, but public schools, and the cloak-and-dagger scenario has been replayed in school districts throughout Colorado. The material being protected so thoroughly has nothing to do with state secrets and is certainly no cause for alarm, but given the gravity with which it’s handled, it’s gaining a reputation as being more precious than gold.

And like the ingots sealed deep in Fort Knox, it’s just as hard to get a glimpse of the Colorado Student Assessment Program unless you’re a student. Not even parents get to see the contents of these annual proficiency exams—not the one being administered this year or last year or the year before. Not ever.

In its 12th year, CSAP is the annual statewide test that measures students’ progress in meeting state content standards. Students in grades 3-10 are tested in the areas of reading, writing and mathematics, and in grades 5, 8 and 10 for science as well. The test is critically important to school districts, which must comply with the federal No Child Left Behind Act, which aims to have all students perform at proficient levels in reading and math by 2014. As a measure of states’ progress toward these goals, school districts are required to make determinations of Adequate Yearly Progress. In Colorado, CSAP is used to determine that progress. Last year, more than 1.5 million CSAP tests were administered to Colorado public school students.

Not even the teachers could see the tests until the day they were given to students. Parents never get to see the test questions. They are also not given to journalists; Fort Collins Now asked for a full copy of last year’s exam in order to view the material and analyze the questions. The request was denied. Not only is the material proprietary to the state but previous versions of the test don’t exist, at least not in the school district. According to protocol, the tests and any extra materials are returned with the same level of security as they are delivered. This is to prevent test questions from being leaked; the vast majority of the material is used on subsequent tests. Security to prevent cheating is paramount. In other words, the CSAP is not a public document even though so much public policy relies on its results. School that perform poorly on CSAP tests risk losing critical state and federal funding in the future.

In preparation for the testing, administrators have spent hours counting and recounting exams, outlining strict rules for administering the exams to prevent anyone from getting an early peek, and aligning themselves with the proper procedures.

“It’s very serious,” said Kim Nohava, principal at Zach Core Knowledge School. “Nobody sees the tests prior to them being picked up and administered. They are kept in the conference room under lock and key with a sign that says the door should not be open at any time. It’s very secure.”

Beth Celva, director of student assessment at the Colorado Department of Education, said CSAP is like any other large scale assessment and that the direction and procedure manual is very clear on how the test must be secured in order to follow the protocol required by the United States Department of Education.

She said that when the CSAP was originally crafted, the CDE and CTB/McGraw-Hill involved thousands of teachers to develop testing items. CTB/McGraw-Hill is one of the leading publishing companies in the testing industry and is contracted by the state to produce and copyright the material. It has also developed tests for other states including North Dakota, New York, Missouri, Florida and California, among others. But some skeptics question the credibility within the testing industry itself.

ML Johnson, Poudre School District board director, said one of the main problems is that the testing industry is unregulated.

“That’s a very significant problem,” he said. “Someone in a closed room sits down and writes questions they think kids should know. ... Anyone can cook up anything.”

The problem is exacerbated in that no one can see the tests to independently assess their validity. Johnson said teachers, parents or district administrators should have access to past exam content considering that the test has so much weight.

The problem is there is no such thing as a “past CSAP test.” Unlike college entrance exams, like ACT or SAT, where all the content changes regularly, CSAP reuses much of the material from year to year.

“The reason past test items are not available is we have variant forms of the tests,” Celva said. “Items change in the test. New items are included in the test.”

However, only a small percentage—about 20 percent—of the exam is changed each year. The CDE used to release items that were no longer going to be used so that parents could see what questions were being asked, but that’s no longer the case. The last time the state released sample test questions was in 2005 and the only items released that year were third- through fourth-grade math questions.

Poudre School District Superintendent Jerry Wilson said the if the state released more items, it would become more expensive to produce the test because it would have to be rewritten more often. On the other hand, he sees the benefit in being able to see what, exactly, is being asked on the CSAP.

“I think there ought to be released items on a regular basis,” he said. “I believe that’s important.”

Chuck DeWayne, PSD director of curriculum, instruction and assessment, said the conundrum is that releasing items is cost prohibitive, but that it would help teachers have a more sophisticated understanding of the expectations. Some PSD schools used to share released items with parents and even construct examples of the exam, so they would know what to expect.

“I’ve proctored the test before,” DeWayne said. “I’ve gone, ‘Wow, I don’t know if I could answer that and I’ve got a couple masters degrees.’”

Though the district does not want to teach to the test, DeWayne said that having a clearer picture of the type of test material would allow teachers to have a better idea what the students were missing and help them figure out why.

Overall though, DeWayne said that CSAP is a well-written test that gives a pretty good reflection of students’ performance. In order to keep it that way, educators maintain that tight security is a must. And therefore, there is a long list of strict rules teachers must adhere to in order to keep the secrets of the CSAP intact.

Teachers are required to check the tests out before testing begins and check them back in when it is over. Trainings trickle down from the CDE to the school districts and then to the teachers. Certain things cannot be hung on the walls and proctors must constantly walk around the classroom. It has even come to the point where students in a particular grade level in a particular school must all take the exam on the same day at the same time. That way kids can’t share what was on the test with others on the playground.

The consequences for not following the protocol can be severe: Last year, Rocky Mountain High School was hit with a negative score to be calculated into its results when an instructor mistakenly allowed some sophomores to use calculators on the math portion of the test. That, and other situations that penalize schools—such as parents opting to take their kids out of the test—led to discussions at the state Capitol about changing the system.

But there are still those who argue that if the test is going to mean anything, it has to remain under tight security and be administered to very high standards.

“I think to have it be a fair test, that’s the way it needs to be,” said Nohava, the Zach Elementary principal.