Looking for the latest stories? We're now at cironline.org

Judging teacher performance is difficult, experts say

The publication of a controversial, and groundbreaking, article by the Los Angeles Times raises complex questions about whether to "out" teachers whose students perform poorly on reading and math tests.

That is especially true when using "value-added" techniques that are complicated even for statisticians who do this kind of thing for a living.

The Times' analysis holds the potential to fling open the door of any California classroom for public examination in a way that has never been attempted before. 

U.S. Secretary of Education Arne Duncan has no problems with the practice. "What do they have to hide?" he said in response to the Times article, referring to the teachers identified in the report.

Bonnie Reiss, Gov. Arnold Schwarzenegger's secretary of education, also gushed. "Publishing this data is not about demonizing teachers," she said. "It's going to create a more marketplace-driven approach to results." 

On the other hand, the United Teachers of Los Angeles denounced the disclosures as "dangerous and irresponsible." Union leader A.J. Duffy is threatening a boycott of the paper, plus possible legal action. 

Let's try to get beyond the rhetoric. 

Researchers I talked with tell me that if this had been an academic study, the researchers would never have been given permission under human subject research guidelines to disclose the names of teachers. 

Jennifer Imazeki, an economist at San Diego State University, wrote on John Fensterwald's The Educated Guess:  

Regardless of how one feels about value-added, as a researcher, I've been shocked at the public disclosure of teachers' names. Most researchers have to sign their lives away in confidentiality agreements if they want to use student-level data with individual identifiers. How in the world did the Times get their hands on this data without such an agreement?

Richard Buddin, a respected economist at the Rand Corporation who as an independent contractor ran the numbers for the L.A. Times, said he had nothing to do with releasing the teachers' names.

In two e-mails to me, he explained that the files he used for his analysis had "scrambled student and teacher identifiers" and that he made "no attempt to link the scrambled identifier with teacher names." "The Los Angeles Times did this after I completed my analysis," he wrote in an email.

So how did the Times get the names of teachers from LAUSD? Simple: They asked for them. 

Robert Alaniz, LAUSD's director of communications, told me the district's legal department concluded that under California's Public Records Act, the district had no choice but to release the names of the teachers, and to link their names to the test scores of their students. He said that if test scores had been used as part of a teacher's performance evaluation, the scores would have remained private. But because they aren't, they are not regarded as confidential information. 

"We vetted it with our legal staff, and determined that the request was valid, and that we did have to turn over the teachers' names," Alaniz said. "As adults, as employees, their names fall into the public domain." 

He said the district has some safety concerns about the Times' plan to publish the names of 6,000 teachers and where they teach, because some may want to keep their location secret from former spouses and others they may have restraining orders against, etc. The district also has concerns about an over-reliance on on using test scores to evaluate teachers. "It should be just one of many different factors," he said. 

All this would be more straightforward if teachers were identified on a clear-cut fact that is either true or false, such as whether they have the proper teaching credentials, or how much they get paid, experts say.

A report issued last month by the U.S. Department of Education's Institute of Education Sciences concluded that "policymakers must carefully consider likely system errors when using value-added estimates to make high stakes decisions regarding educators."

And last fall, the National Research Council took a close look at the administration's promotion of the value-added methodology as a criterion for states to qualify for its $4.3 billion "Race to the Top" program. 

The headline announcing its report, referred to briefly in the Times article, declared, "Value-added methods to assess teachers not ready for use in high-stakes decisions."

The distinguished panel that drew up the report, which included two UC Berkeley professors, Michael Hout and Mark Wilson, warned the administration that "although the idea has intuitive appeal, a great deal is unknown about the potential and the limitations of alternative statistical models for evaluating teacher's value-added contributions to student learning." 

One of the concerns raised by the panel was the complexity of the statistical methods used, which would make "transparency" difficult and critiquing an impossibility for anyone but the most sophisticated statistician. 

That seems to apply to the dense report written by Buddin accompanying the Times article, in which he explains his methodology. 

Take this paragraph, picked more or less at random:

Data sets on teacher inputs are incomplete, and observed-teacher inputs may be chosen endogenously with respect to the unobserved-teacher inputs (teacher-unobserved heterogeneity). For example, teacher effort may be difficult to measure, and effort might be related to measured teacher qualifications, i.e., teachers with higher licensure test scores may regress to the mean with lower effort.

Or this paragraph: 

Teacher heterogeneity (φj) is probably correlated with observable student and teacher characteristics (e.g., non-random assignment of students to teachers). Therefore, random-effect methods are inconsistent, and the fixed-teacher effects are estimated in the model. The fixed-teacher effects are defined as ψj=φj+qjρ.

It will require a lot more than fifth grade arithmetic to penetrate that algebraic thicket. 

UPDATE: From Jason Felch:

We welcome the discussion and scrutiny of our articles and methodology, but several things in Freedberg's post require clarification. ...

The premise of Freedberg's post is that, had it been an academic publication, it may have been done differently. It was not an academic publication, it was investigative reporting done in the public interest with public records. The decision to post teachers names was a journalistic one made after careful consideration at the highest levels of the LA Times.

Freedberg refers to the value-added technique as "mostly untested." The technique was developed in the 1970s and has been used by school districts and states since the early 1990s. Dozens of academic papers during that period have studied the approach, its potential and its limitations. In 2008, for example, leading researchers from Harvard and Dartmouth conducted a random-assignment experimental validation of the approach with LAUSD data and published their results here: http://isites.harvard.edu/fs/docs/icb.topic245006.files/Kane_Staiger_3-1... (PDF)

Finally, Freedburg takes the curious tack of criticizing our transparency. The methodological paper he cites is written for the research community so that our approach can be vetted by experts in the field. For a lay audience, the Times has also published a lengthy Q&A, two videos explaining the approach, a list of research papers reviewed by reporters, an About this Story explaining our process and an Editor's Note addressing the decision to publish teachers names. It appears Freedberg did not see, or did not care to mention, those efforts.


Filed under: K–12, Daily Report


Comments are closed for this story.
Caroline Grannan's picture
Louis Freedberg shows that journalistic standards live, just when I was despairing.
Caroline Grannan's picture
Also, I would love to get the L.A. Times reporters to explain exactly what those quotes from Buddin's report mean. Bet a $100 donation to the PTA that they couldn't even begin to try. Based on many years of editing reporters' copy, I predict a "that's what it said..." response.
cajacobson's picture
I've already created my Scarlet "T". Think I'll suggest it to UTLA. Perhaps if teachers start wearing them someone might get the metaphor - or at least know who to cast stones at. Although they'd probably claim that martyrdom is unbecoming a "professional" and we are just whining again, you know, against children and for the status quo and all that...
Arkady Akademos's picture

The LA Times should be deeply ashamed. LA and DC are completely backwards on education. It is appalling. Stupidity is no excuse, and the possible ulterior motives are grotesque and criminal.

This extreme pressure on teachers to add value is stupid on the stupid face of it. The key problems in education are not underdeveloped teachers, though some of it involves teachers unable to work effectively in insane situations re resources, overcrowding, administrators, policies, etc. And, of course, there will always be the occasional incompetent or insubordinate teacher, but value-added nonsense is not the best way to weed them out. In fact, it may not work at all!

The majority of the problems reside in overall school missions, methods, resources, curricula, and administrations; lack of parental involvement or family dysfunction; blighted communities and chronic student issues that affect focus, motivation, or self-perception.

Yes, teachers are the main masters of adding value, but the real problems are the disaster areas in students’ lives, academic and real-world. That’s when their learning stalls and skills fade. That’s what has to be fixed. And to suggest that it be fixed by making everyone a super-excellent teacher to double (or triple, if need be) the learning pace is unrealistic, unfair, and evasive. Plus, this whole value-added concept has some inherent problems, besides our inability to accurately measure it. Ever heard of learning curves? Developing humans take them in different ways in different areas. Sometimes a major block, or steep bump, is natural, yet will require serious parental intervention, like a private tutor, when school intervention would be way too little, way too late. And there are plenty more nonlinear aspects to learning in all stages of our development: plateaus, epiphanies, synergies, stases, setbacks, etc.

These deformers really undo themselves. This is very convincing of their lack of education. They should read a book, Diane Ravitch's Death and Life of Public Education, learn something, stop the harassment and nonsense, and get out of office and out of sight. The gig is up.

aby1's picture
Mr. Felch, Please go to work as a teacher in a poor neighborhood in LA. Then let us rate you by your own standards. And of course, even though you would be teaching unruly children, let's publish your students test scores. What's good for the goose is good for the gander!
aby1's picture
So Mr. Felch, When are you and your comrades in arms going to go teach in an inner city school? What - did you say "Never!" When are you and your comrades in arms say you were going to volunteer to tutor in an after school program in an inner city school? What - did you say "Never!" When are you and your comrades in arms say you were going to help disadvantaged kids in a reading program? What - did you say "Never!" When are you and your comrades in arms going to attack teachers in schools you would never lift a finger to help? What - you said "Right Now!"
Caroline Grannan's picture
With the newspaper industry collapsing, a number of still-employed newsroom employees MAY wind up displaced and career-changing into teaching, as happened in my own family. It would only be poetic justice in some cases.
Norm Scott's picture
I prefer the Taliban method of stoning low scoring teachers to death.
AirBeagle's picture

Mr. Felch, your whole premise is flawed and disingenuous. As I wrote the Times, your article is an irresponsible, disingenuous & truly immoral piece of propaganda, based solely on your own unprofessional and uneducated "analysis" of on ONE man's "research." It is nothing more than a press release served up as earnest journalism. But there is no journalism here; there is just the crude & shameful public scapegoating of public school teachers in service of a corporate, for-profit ideology. Shame on you, Mr. Felch, and on your colleagues in this hit piece, Jason Song and Doug Smith.

The sole source for your piece is Richard Buddin, a RAND Corporation hack. He has been employed by RAND since 1974 and has never held another job with any other company. He has been a "dissertation advisor," and an economics professor at UCLA and the RAND grad school of Santa Monica. Buddin has never held a public education job; never taught elementary or secondary school; never been a teacher. You can claim he's an "independent contractor" all you want; he's never worked for anyone other than RAND and the two cannot be separated, your wishful thinking notwithstanding.

And so this is what you think is "analysis"? A RAND propaganda study without any questioning on your part of this so-called "value-added" nonsense being perpetrated on the nation's education system?

Yes, I'm a former California public school educator currently teaching in Tennessee. I'm also a former journalist. I recognize this junk for what it is: A thoroughly dark day in American journalism. And an extremely sad and damaging day for LA's public school servants.

teach234's picture
Were LAUSD teachers alerted, when they were hired by LAUSD, that the value added test scores, along with their name and photo ID would be published in the LA Times? The fact that the LAUSD school district, deemed it necessary to publish results, along with teacher names and photos, with the results of the one-two hour value added tests that occurred, in 2004-2007, is a symptom of a school district that believes it is okay to humiliate teachers, and operate with hidden agendas. I have to wonder what other atrocities the LAUSD district is inflicting upon it's teachers as a matter of practice. Where are the LAUSD statistics for the the proportion of teachers hired with no credential or intern or emergency credentials, teacher salaries, district hiring, firing and training practices, teacher performance rating practices, classroom leveling practices, team teacher practices and intervention policies, classroom distributions of special education, regular and honor students mixes in the classrooms, and teacher retention statistics. These statistics, in my opinion, are far more relevant with respect to teacher performance, than the value added tests that occurred over the duration of several hours, at most. Further, what were the student grade histories in comparison of John Smith and the other teacher? How about the distribution of special education, regular education and honor students, and migrant students in each classroom? Does team teaching ever happen in either of these classrooms? If so, how often? Does Mr. Smith and Mr. Aguilar take turns observing each other's classrooms? What is the difference in salaries of the two teachers? What were the behavior histories of the students in each classroom? Does Mr. Smith feel supported by his principal and administration? Are teachers hired with known language barriers, ie family language and classroom language differences? Is there a difference between resources made available to both teacher, is there a difference in the appearances of desks, black board or white boards, number of teacher assistants, etc, etc. These statistics could be easily gleaned, yet appear to be completely ignored in the “value added” ranking that the LAUSD district appears to be hiding behind and hoping to be completely unaccountable for it's unbelievably horrible treatment of it's teachers, except for Mr. Aguilar. Teachers are tested, the tests are called the CSET and CBEST. I passed both tests, but I failed the test in knowing what horrible thing LAUSD is going to do next to it's teachers.
cypherpu's picture
A view from outside... I am a parent in California with 3 children enrolled in public schools. I have a strong interest in school performance. I have a strong background in statistics and heathcare. It was with great interest I read the LA Times article. No matter what happens between the UTLA and the LA Times, the direction of change in education is inexorable: more and more, teachers and schools will be assessed quantitatively for their effectiveness. In healthcare, this has been going on for a number of years. For instance, if a physician settles a lawsuit for more than $30,000, his/her name is published by the Medical Board of California. If a physician is sued in many cities, their name is published in the local paper. There is a national database which contains the name of every physician who has settled a lawsuit for even $1. There are multiple websites dedicated to evaluating patient's experiences with physicians. Cardiac surgeons mortality statistics are published yearly by many states. Would anyone say that this is "naming and shaming"? Probably not, because we attach a high importance to the delivery of quality healthcare. Similarly, more and more parents are understanding the importance of quality education, and their standards and demands are rising. Members of the UTLA can certainly respond to this study with anger. Or, they might be thankful that they cannot sued for their sometimes poor outcomes, as can health care professionals. Either way, better to focus money and energy on guiding development of teacher evaluation guidelines than trying to impede or slow development. Finally, I have looked over the statistical methodology used in the LA Times article. It certainly is not fool-proof, but it is not unreasonable, and it suggests areas where educators can improve educational outcomes.
David B. Cohen's picture
Would you then argue that you can look at a mortality rate of two surgeons and conclude that one is better than the other? Don't you have to know something about the patients, the hospitals, the assisting doctors and nurses, post-surgical care, and other factors? Sorry, no dice. You can't reach meaningful conclusions on limited data. You can release the data and hope no one misuses it, but that's sort of like leaving matches out and hoping your kids don't light something. Read Daniel Pink's "Drive" and see the volumes and volumes of studies that indicate narrow measures, high stakes, and negative consequences will actually lead to diminished performance.
martha.infante's picture
"Understanding the importance of quality education" has not been accomplished with the piece written in the LA Times. What has been accomplished is a simplistic and erroneous definition of teacher effectiveness. The reporters defined teacher effectiveness solely by the value-added score. As a teacher and parent, trust me, that is not how I evaluate my own child's teachers. For quality education I want to see a well-rounded curriculum, project-based learning, fostering a love of learning, and providing a safe environment for academic and social growth.
David B. Cohen's picture
Mr. Felch offers one study as support, but does not refute the larger accusation: he, his partners, and the Los Angeles Times have engaged in a teacher evaluation practice that is irresponsible according to the accepted guidelines of all the major educational research organizations.
teach234's picture
Teaching at a high risk school I learned the non-random process of how students are assigned to classrooms, which will effect the outcomes of value added metrics. Favoritism, with respect to principal and teacher relationships, exists in education. Favored teachers are assigned better resources, air conditioned classrooms, a less diverse student population, and pretty much get whatever they want, while the less favored teachers pick up the slack. These scenarios in K12 education, happens all the time, and this favoritism effects value added metrics.
bpeterson1931's picture
Secretary Duncan said in a June 2009 speech, “Just as the American Bar Association polices the legal community and the AMA (American Medical Association) does the same for the medical profession, you must get more serious about accountability.” What are the ramifications of this statement? One can imagine that in the last few years many lawyers working as LA public defenders on cases from the south central district have not had optimum outcomes. If we could develop a list of name of those public defenders that have had non-optimum outcomes over, say, 50% of the time, should we publish those names? How about doctors with similar outcomes handling alcohol and drug cases---publish their names? If the community did not want suffer some high legal cost down the line it would be advisable to proceed with caution when publishing the names of lawyers or doctors. Michelle Rhee, Chancellor of D. C. Public Schools, has recently fired a number of teachers which her system has determined to be “ineffective.” If we are following the guidance of the ABA and the AMA we can assume that provisions are being made for the appropriate teacher’s professional organization to hold hearing in these cases. Naturally, if these teachers are shown to be ineffective their teaching certificates should be immediately rescinded. We would not want ineffective teachers anywhere in the system. Using the ABA/AMA model we would assume that a formal hearing has been held for each teacher involved, the applicable teaching certificates have been revoked, and that there is an appropriate appeal procedure (I would be surprised if this is the case).
ink109's picture
Naturally, if these teachers are shown to be ineffective their teaching certificates should be immediately rescinded. stock market today
masini's picture
I really do not know if I could do this. A teacher should be judged by the finished product, that education provides students the time this is very difficult. Children have different natures and abilities so very different. They may perform in a given area if the teacher that knows how to guide it. This is the true role of the teacher. But getting into education and other factors such as family, society. And they have a decisive influence on children's future. If the teacher can not be harmonized to be very good, the results will not occur.cheap auto parts

via Twitter

© 2013 California Watch   /  development:  Happy Snowman Tech   /  design: