THOMPSON: DEPARTMENT OF EDUCATION GETS FAILING GRADE ON TEST SCORE OVERSIGHT
- Department offers "dog-ate-my-homework" excuses for missing monitor reports-
- "DOE has created an environment that both encourages cheating and allows the Mayor to claim achievements that cannot be verified," Comptroller says-
The New York City Department of Education (DOE) lacks proper controls over English and Math testing, and standardized exams are exposed to potential cheating and test manipulation, New York City Comptroller William C. Thompson, Jr. charges in a new audit.
"The Department of Education boasts that rising tests scores prove that the achievement gap is closing in our city's public schools and that
The audit - available at www.comptroller.nyc.gov - covered English Language Arts (ELA) and Math tests for students in third, fourth and fifth grades during the 2007-2008 academic year. Thompson's audit exposes glaring flaws that illustrate woefully inadequate monitoring of the administration of standardized tests.
"If standardized testing is considered the benchmark by which the Department of Education and the Mayor measure student achievement, then the City must ensure that their efforts are both transparent and verifiable," Thompson said, adding "We must make every effort to ensure that test scores accurately reflect students' educational progress."
Among the audit's findings:
- The Department failed to keep track of monitors who were required to be at schools and complete mandatory checklists to ensure test integrity. Often, those monitors showed up late, or didn't show up at all.
- Monitors indicated they oversaw certain activities that they could not possibly have been able to monitor. Seventy one percent of monitoring forms indicated "yes" to questions that were impossible for monitors to answer.
- The Department stopped performing data trend or erasure analysis to identify possible cheating or testing irregularities in 2002.
- Portions of tests remain at schools for many days, increasing the risk of being inappropriately manipulated.
The DOE provides primary and secondary education to more than 1 million pre-kindergarten to grade 12 students in more than 1,400 schools. The Department seeks to teach students to meet grade-level standards in reading, writing, and math, and then tests students to determine how well they are meeting these mandated learning standards.
Students in grades 3 through 8 take both the State standardized ELA and Math tests. The ELA is a timed test containing multiple-choice questions and performance items, and it is given over a three-day period in grade 4 and two-day period in grades 3 and 5. The Math test consists of two or three test sections depending on grade level; the first section of the test is given on Day One, and the second and third on Days Two and Three.
Auditors found that the Department does not keep track of monitors assigned to visit schools or their submission of mandatory checklists. Thompson's office did not receive monitoring checklists for nearly half (48%) of the monitors' scheduled visits it requested for the audit. The DOE could not determine whether monitors were present at schools for which there were no checklists or whether monitors visited schools but checklists were somehow missing.
"My office raised a number of questions about inconclusive records of monitor visits, checklists, and reporting of testing irregularities, and the Department offered excuses equivalent to 'the dog ate my homework,' even including explanations that monitors have other duties and often have to cancel because of other responsibilities," Thompson said.
Additionally, monitors did not always ensure that test administration procedures were followed. Monitors are required to arrive at assigned schools by 7:30 AM to ensure that test booklets are still in shrink-wrapped packaging and secured in a locked place.
However, auditors found that 32% of monitors arrived at the schools after that time; 16% arrived at or after 8:00 AM; and, one arrived as late as 10:20 AM.
Furthermore, the Department's checklists ask specific questions that require monitors to observe classrooms while tests are in progress. Auditors noted that some monitors did not observe classrooms while tests were in progress - so it would have been impossible for them to legitimately complete checklists reflecting that they did.
"The presence of monitors can be an effective tool used to minimize the potential for improper conduct," Thompson said. "Since monitors are an important part of the DOE's efforts to identify testing irregularities, it is vital that it track monitoring activities and carefully review the submitted checklists."
Auditors further found that the DOE stopped performing data trend or erasure analysis to identify possible cheating or testing irregularities. Data trend analysis, although not required by the New York State Education Department (NYSED), would allow the DOE to strengthen internal controls in the areas where weaknesses are identified and help the DOE in targeting the schools that should be monitored, rather than relying on its current practice of randomly selecting schools.
The DOE officials said erasure analysis is used when appropriate during the course of specific investigations of testing irregularities. However, since the Department couldn't provide any supporting evidence, auditors couldn't ascertain the extent to which erasure analysis is currently being used.
"The DOE has created an environment that encourages cheating and allows the Mayor to claim unverifiable successes. The incentive to manipulate test scores, and the risk that this may occur, has increased in recent years, particularly as the Mayor continues to emphasize the DOE's supposed success," Thompson said. "In such an environment, tools such as erasure analysis are needed to help the DOE more effectively identify possible testing irregularities."
Thompson's audit further noted that the Day Two and Day Three portions of the ELA and Math exams remain at schools for a number of days, increasing the risk of manipulation. After completion of the Day One sections, schools are required to deliver the student-answer documents to the Integrated Service Center (ISC) in the school's borough by 3:00 PM. However, the Day Two and Day Three sections are maintained at the individual schools until the completion of make-up exams - up to six or seven days from the initial administration of the exam.
Because the tests are maintained at the schools for many days, there is an opportunity for answers to be reviewed and changed from incorrect to correct answers, Thompson said, adding that school officials can review the tests and return them to students who did poorly so that answers can be revised.
During the course of the audit, Thompson's office attempted, but was unable, to track the late delivery of answer documents to the ISCs, as well as the number of make-up exams taken by schools over the last few years. The Department maintains this information for only one year from the exam date.
Thompson's office also discovered that the Department doesn't have a formalized process in place to ensure substantiated allegations of cheating are shared with the Office of Accountability (OA), which is primarily responsible for ensuring that test administration is in compliance with City and State guidelines.
Thompson's review of 13 allegations substantiated by the Office of Special Investigations (OSI) from 2006 - 2008 revealed that nine - or 69% - were instances of improper proctoring in the classroom by school officials. In addition, there were instances where the tests were easily accessible because they were left in an unsecured location.
"Since improper proctoring was identified as a significant issue during the course of these investigations, the OA needs to develop controls to address it," Thompson said.
Overall, the audit found that the DOE has adequate controls with respect to ensuring that schools are familiar with established procedures when administering State standardized tests at elementary schools. Additionally, schools visited by auditors generally complied with
As a result of the audit Thompson made 14 recommendations, including that the DOE should:
- Accurately track the assignment of testing monitors.
- Coordinate with the NYSED to obtain test answer keys promptly following the administration of each test thereby enabling the DOE to perform a timely erasure analysis. However, the DOE should perform erasure analysis to identify possible improprieties regardless of when it receives the answer key.
- Compile, maintain, and track data on the number of make-up exams that are taken for the Day Two and Day Three ELA and Math exams.
- Identify indicators to detect unusual patters that may indicate test tampering or irregularities and collect sufficient data to adequately track those indicators. Based on the information collected, the DOE should target those schools with unusual patterns for further follow-up.
- Ensure that the OSI formalizes a process ensuring that all instances of substantiated cheating are shared with the OA, so that the OA can strengthen existing controls or develop new ones in an effort to prevent the occurrence of cheating.
###