Problems Undercut CLAS Tests, Panel Says : Education: Statistics experts say sampling and scoring procedures need to be improved, and recommend that an outside firm be hired to administer the exams. But they praise program for innovation.

Budget constraints, scoring glitches and the difficulties inherent in such a complex new testing system have undercut the pioneering California Learning Assessment System exams, a panel reviewing the CLAS tests’ first rocky year has concluded.

Included in the panel’s 62-page report, to be released today, was a recommendation that the state Department of Education hire an outside firm to administer the tests. The group also called for the department to turn over responsibility for scoring and interpretation of the results to a statistics expert and to hire a full-time technical coordinator to oversee the effort.

The panel also cautioned against releasing individual student scores, scheduled to begin this year, until the bugs are worked out and parents and teachers are thoroughly briefed on how to interpret the results. The governor’s office has been pushing for individual results so parents and teachers can gauge students’ progress.

The exams, intended to pinpoint a school’s academic strengths and weaknesses by assessing how well students can think and write about what they have learned, were given for the first time last year to about 1 million public school students in fourth, eighth and 10th grades. To save money, the department decided to score only some of the exams in reading, writing and mathematics before releasing results for each school.

Almost immediately, questions surfaced about the accuracy of the scores, which were widely published and used by parents and others to evaluate schools. A Times analysis showed samples in some cases were so small that results for an entire school were based on a handful of graded test booklets.

The panel of three statistical experts, appointed by the department in April to review its scoring procedures, found that there were serious shortfalls in the numbers of 1993 exams graded in about 250, or 3%, of schools. In those cases, additional exams were scored and results adjusted, the Department of Education said.

But the experts, headed by Lee J. Cronbach of Stanford University, concluded that “few school-level reports in 1993 had adequate reliability.”

CLAS developers “made unreasonably optimistic estimates as to the accuracy that the resulting school reports would have,” Cronbach wrote in a letter accompanying the report.

Problems inherent in the sampling procedures were compounded by lost test booklets, errors matching student identification sheets to the booklets, mistakes by the contractors helping administer and score the tests, and the rush to get scoring done and results released before the 1994 round of testing began in the spring.

“Inadequate review under time pressure led to the release of some reports that were manifestly untrustworthy,” Cronbach added.

To make matters worse, many parents and school staffs had expected all the tests to be graded, the report noted.

“CLAS laid a trap for itself when, in the public information packet intended to develop understanding of the system, it omitted mention of its intent to score only a fraction of booklets,” the report said.

The panel recommended careful tracking of student participation in the 1994 test, noting that parent protests over test content resulted in widespread boycotting at some schools and may have skewed the results.

Although the panel was assigned to look primarily at the sampling procedures, it also noted difficulties in scoring the open-ended, or “performance-based” questions that are the hallmark of CLAS and other pioneering testing systems around the country. Instead of merely selecting the correct answer from a list of possibilities, students are being asked to solve math problems and explain their methods or read a passage from literature and write about their responses.

But the panel, which also included Norman M. Bradburn of the University of Chicago and Daniel G. Horvitz of the National Institutes for the Statistical Sciences, commended CLAS officials for some of their work.

“CLAS and its contractors have embarked on an unprecedented and imaginative project . . . and they have done many things well,” Cronbach wrote, adding the 1993 efforts were “highly successful as a trial” and shed valuable light on many aspects of the just-emerging field of performance-based assessment.

“All the shortcomings of CLAS 1993 can be remedied, within the inexorable limits of the time available for testing and the costs of scoring,” Cronbach added. “CLAS, as it matures, should be able to deliver a highly useful product.”

Acting state Supt. of Public Instruction William D. Dawson said he welcomed the report and has already acted on some of its recommendations. Scoring of this year’s tests, done by specially trained teachers at sites around the state, already is under way and reflects the panel’s suggestions, Dawson said.

The report comes as CLAS, which is also under attack from religious conservatives for the contents of its reading and writing tests, faces an uncertain future. Next week, legislators are scheduled to work on a compromise bill that would continue the system--renamed the California Comprehensive Testing Program in state budget sessions this summer--while addressing many of the criticisms.

Gov. Pete Wilson’s education adviser, Maureen DiMarco, who has been critical of the way the testing has been developed and carried out, said the report “not only validates our concerns and the concerns of others but also gives us even greater impetus to continue to work to revise the assessment system.”

The governor’s insistence on individual scores could be met by adding more traditional, skills-measuring methods, such as multiple-choice questions, she added, noting some of the revisions sought by her boss and other CLAS critics already have been added to the proposed legislation.