Boosting School Accountability Transcript

|

MR. MCMAHON: Good morning ladies and gentlemen.  Thank you for coming. My name is E.J. McMahon, I’m a senior fellow with the Manhattan Institute and Director of the Institute’s Empire Center for Public Policy. And on behalf of the Empire Center and our co-sponsoring organization, the Foundation for Education Reform, I want to welcome you to this forum today. And before going on, I want to also express our thanks to Senate Minority Leader Malcolm Smith and the Senate Minority Conference for allowing us to use this wonderful room and we thank them very, very much for that.

As you know, Governor Spitzer has proposed an education reform package that includes much more money along with expanded choices for parents, and a strong added emphasis on accountability. Now, greater accountability in education is certainly something everybody here can support, but how do we define it? What do we mean when we say accountability and how will it be actually implemented? That in a nutshell is the question we want to explore today and we have some distinguished speakers, we think, who can help us do that.

I’d like to begin by introducing as our first speaker for welcoming remarks, John C. Reid. And Mr. Reid, and if you don’t know him, is the newly appointed Assistant Secretary for Education in Governor Spitzer’s office, where he’ll be working closely with the governor’s Deputy Secretary Manny Rivera to advance the governor’s agenda for education reform and accountability.

Mr. Reid is a former senior vice president of Coca-Cola and he was chief administrative officer of the City of Atlanta. But since 1995 he has been deeply involved in education issues, both at the policy and at the operational level. He has served as the president of the Edison Project and Learn Now and serves on the boards of the Princeton Review and Tutor.com. In addition, he has worked extensively on issues involving teacher recruitment and retention, both as a consultant and as an employee of the New York City Department of Education. Mr. Reid holds a bachelor’s degree from Brandeis University and a master’s degree from MIT. Please join me in welcoming John Reid.

MR. REID: Good morning everybody. And thank you E.J. and thanks to Tom Carroll for allowing me to understand that this event was taking place and to understand the importance of the event and it’s a pleasure to be here. I don’t believe that anybody in the room would think it was an understatement to indicate to you that this is a momentous time to be at the state Capitol talking about this issue. And in particular, I don’t think that anybody in the room would argue the thought that the timing of a forum such as this is rather impeccable. You couldn’t be more right on time. There is a recent editorial that I’m fond of that said, “Imagine after years of the usual course of good government activities demanding reform and politicians half-heartedly promising to make it happen, the rules and the culture of Albany really did change.”

So, what would happen?  People like you and I from both sides of the aisle and from all persuasions would make a race to be a part of the dialogue and I’ve done that and you’ve done that and it’s great. I suppose it’s fair to say that virtually all of us in the room have waited patiently for a moment in time like this to come where we all can engage together in making this right and to make it work for the entire state.

It’s clear to me that? you have an ambitious agenda this morning and I don’t want to get too much in front of it.  I just simply want to say that it appears to us, that it fits perfectly with the governor’s agenda, and that obviously is to take accountability in all of its myriad shapes and sizes to a new level in order to get this right.

So, we’re moving investment levels to historic levels. And the governor’s made it clear that he’s willing to put the resources of the state firmly and dramatically behind the most ambitious education agenda in the state’s history. But, and this is the big but, he is only willing to do that if that reform agenda is accompanied by new and major levels of accountability?

From a financial accountability standpoint he has made it clear that he wants the dollars to be tracked so that the citizens of the state know where the dollars are going. Pure and simple.  And he wants fundamentally to take all the excuses away, so that there’s no more excuses for failure. He’s made that clear a number of times and he couldn’t be more serious about it.

The magnitude of the dollars is stunning. I think you all know that this year’s budget increment is $1.4 billion and I don’t think I need to tell this room that an 8% increase in a budget of this magnitude is very significant.  So the idea that it would be coupled with accountability is essential. I think there’s no one that would disagree with that.

So, along with financial accountability comes this second idea of programmatic accountability. And the governor has made it abundantly clear that we’ve advanced as we move in to the 21st century to a level now where we are really beginning to know what works.  And wouldn’t it be a shame if those massive incremental dollars would be spent on areas where it wasn’t a clear-cut case for what works. And he simply said, let’s put the money against proven programs? You know, now there’s a big idea. Smaller class sizes, a little bit more controversial but not much. Middle-school, (and) high-school reform, teacher quality, all of these things are the direction that the governor is taking and it’s all in the spirit of programmatic accountability whereby the people who get these funds will be responsible for channeling them in to programs that work.

And then there’s a third. Financial accountability, programmatic accountability and the final one is, kind of, the tough one, it’s where the rubber hits the road and that is performance accountability. It’s the one that makes certain people say “ouch”. And it’s a tough nut and I think that this group understands that it’s a tough nut, this is a very sophisticated audience.

From the governor’s standpoint he’s made it clear that if children fail, adults must be held accountable. He said it purely and simply. And his contract for excellence, the idea behind that is to make sure that everyone is implementing a plan, that the plan is being implemented with programmatic resources that are targeted in the right way, but, and this is where performance accountability comes in, but where there is a celebration and recognition of success, which is co-equally important with the notion that there is a consequence, for failure. It is the idea that failure matters. And success matters, and that we somehow will get dramatically better at recognizing those two ideas in ways that can actually be implemented statewide.

The governor has no illusions, whatsoever. He is very clear that this will not be easy; he knows that this is just the beginning of lots of discussions. He welcomes the learning and the debate. And all of the ideas that will come forth.

I was just with 48 superintendents, all of whom have local consequences of this very massive, strategic policy change at the state level. And he welcomes that debate, about how exactly this is going to be accomplished. And, of course, that’s where this forum comes in. He’s made it perfectly clear that he’s willing to discuss it all, as long as there is no compromise on its fundamental premise that this educational system needs to be reformed and it needs to rise to the challenge of the 21st century.

He appreciates you, he appreciates the energy and the caring that you bring to this issue of accountability, and he and all of us wish you the very best for a productive and successful exchange of these extremely important ideas.
Thank you very much.

MR. MCMAHON: Thank you very much John and we certainly all wish you all the best in your new position.

As our next speaker to give a presentation on this issue today, we have our co-sponsor, the president of our co-sponsoring organization, Tom Carroll of the Foundation for Education Reform & Accountability. Tom also is the chairman of the Brighter Choice Charter School for Girls, and the Brighter Choice Charter School for Boys and he serves as chairman of A Brighter Choice Scholarships – an innovative, privately funded initiative that provides scholarships for inner city children to attend the private school of their choice.

He previously held research, fiscal and administrative positions in New York with the governor’s office, and the state Legislature in the Division of the Budget. Mr. Carroll’s also a member of the Board of Advisors for the Washington DC based Center for Education Reform and the Graduate Management Institute of Union College of Schenectady.  Thank you very much, Tom Carroll.

MR. TOM CARROLL:  I agree with John Reid, that this is a historic moment in New York state, and although I don’t share the governor’s belief in the value of all the extra money he’s putting up and where that will lead, I think there is a consensus that if that decision is being made, and it is, I think it is one that is broadly agreed upon among most of the key stakeholders, that it has to be matched with accountability. And the governor in his visionary speech at Chancellor’s Hall across the street included, among many nice lines, including the following: “There will be no more excuses for failure. The debate will no longer be about money, but about performance. The goal will no longer be about adequacy, but excellence and the timetable will no longer be tomorrow, but today.”

I share that sentiment and I think what now needs to happen, taking the vision that the governor articulated, is there are a tremendous number of operational details that have to be thought through in order to make this vision a reality over the next few years.  The governor has hired able people in the form of Manny Rivera and John Reid to begin that process, but we are indeed at the beginning of that process. And so the vision for this conference was to bring together, to start a dialogue which we think will take, will continue for a number of years among the key stakeholders to identify what the key issues are that have to be considered, and what the key concerns are and what the areas of consensus and disagreement might be.

And we think, from my perspective, there are a couple of things that we think are very important considerations to consider, and we’ve released a report today which is available in the room for those who haven’t already grabbed it. The first thing is the importance of measuring the right stuff and measuring it well. And what I mean by that is, right now in New York State, we do these snapshot tests every year and what that test will tell you is how that particular group in a particular grade is doing at that particular time.

People make the mistake of looking at one year’s snapshot to the next and assuming that the ups or downs tell you anything about whether an individual school or school district is getting better or worse. And frankly since there are different groups of kids being tested each year, and there’s no adjustment to follow the same kids over time, you have the problem? of having a lot of false positives.

A lot of schools are identified as bad that aren’t necessarily bad, and you also have good schools that are identified. You have it working in both ways, good schools identified as bad, and bad schools identified as good by the simple fact that the data does not correctly measure or seek to measure something more important, which is the gains in the same students over time. I think that is the most fundamental change in the architecture of the state testing system that needs to occur in order for any of the governor’s reforms to work.

Similarly, if you’re going to hold schools accountable, and get people to buy into that, people have to agree that the fundamental way that students are measured in the state actually is legitimate and meaningful, or otherwise there’ll be no credibility to the system.

The second point is I think there’s a need to make accountability transparent and understandable. For accountability to be real, it requires that the public and policymakers – people that spend their lives thinking about psychometrics – that they actually have a sense of what’s going on.

I personally think that the current system of doing level one, level two, level three, level four is not understandable. Being involved in a number of schools in Albany, I can assure you that there are virtually no parents who understand this system.

So there is a need to go to a scale if you will, even if we administer the same exams, that’s more understandable. Whether it’s an A/F system, or 0/100 system or what have you, but when you do a 1, 2, 3, 4 system and American culture is deeply embedded in everybody chanting they want to be number one, people don’t understand that it’s actually better in a state test to be number four and not number one. I think it causes unnecessary confusion.

The third point is, there needs to be a fair and more accurate labeling of schools. The school-in-need-of-improvement designation falls on schools equally if they miss one or two of their sub-group targets, or they miss 30 or 40 targets, and it is simply not accurate to take a school that basically is doing a really good job and failing with a particular subgroup and labeling them the same as the school that frankly is in total meltdown.

So part of this relates to the way NCLB is structured, but right now, literally, there’s more than 50 different ways that a school district could end up being listed as a school in need of improvement, based on when you count all the different subgroups, all the grades in all the subjects, and I think there needs to be a more balanced way, a more nuanced way and a more accurate way for the public to understand, to sort out what schools are good, what schools are mediocre and what schools are bad. And the current system doesn’t do that, and I think some of the districts have rightly complained that relatively high performing good districts have wrongly been labeled and kind of the parlance of newspapers as failing schools when they actually are not.

Fourth, there needs to be a serious discussion about eliminating what I would call conflicts of interest, which is right now, although? people, virtually everybody in this room would know this, but most people on the street don’t know it, school districts are allowed to score their own exams. And so the question is when we move to assist them, and which is not just reporting results, but there actually is, John Reid indicated, that there are going to be consequences for failure. The temptation and the pressure to shave at the margins, how very subjective portions of state exams, particularly the fourth grade writing exam, the Rugerts are so vague, there’s any number of ways you could score that exam, depending on who’s looking at it. Two reasonable people could come up with completely different scores.

So given that there is some subjectivity, if you pair that with actually an incentive for grading the exams, I think it’s extremely dangerous as we move forward. Other states contract it out, other states require that the scoring be done regionally or be done by qualified professionals in other areas of the state, kind of swapping tests back and forth.  But certainly, like in the case of Albany, where the Albany district scores its own exams, I think it’s not, I don’t think that’s a smart way to go and will lead to a loss of public faith in the system going forward.

Fifth, there’s a need to ensure timely reporting. This last year we had our first graduating class at Brighter Choice, we’re a K4 school. We took the English exam in January, we got the results in September, those students graduated in July. So we were glad to find out that they scored well, but to find that information out was not useful to educators in the building at all, because there was nothing we could do, there was no useful information that came out of that that we could use in the classroom.

So there’s a need, although the state testing system serves multiple purposes, primarily an accountability one, if you’re going to administer the test, it would make a lot more sense to do it on a timely basis.
Now this year under fairly intense criticism and pressure, the department has accelerated the timetable, and they’re attempting with more resources to get it done earlier. And that’s an impulse that I support and I think we all should encourage, and support giving the department whatever resources they need in order to chop down the turnaround time for state exam reporting.

Six, I think there needs to be, even if you have a system that correctly measures and correctly reports, there needs to be actual consequences at the end of the day for what happens. Paul Peterson will be talking in a little bit about accountability systems in other states, including Florida for example. But that’s a state where, if the schools are doing well, they get additional resources, and if they’re doing poorly, the parents whose kids are trapped in low-performing schools are given a greater array of options to go to other alternatives including in that state, scholarship or voucher programs.

Now that’s been truncated by the court system in Florida, but I think the system of expanding options for kids, particularly in the lowest performing schools, needs to be part of an accountability system. So I think these are some general, broad principals. We’ve had some preliminary conversations with some of the key stakeholders, and I think it will be interesting to see as we hear the presentations as we go along, whether what areas we can find consensus on as we move forward in terms of restructuring the state system. I found very few people that are enamored of the current approach of accountability in New York, and I think the governor has raised the bar in terms of providing a vision of where we need to be, and I think the challenge to all the people in the room, and the challenge for the administration, the legislature and the Board of Regents, is taking that vision and operationalizing, and figuring out where the rubber meets the road, how can we actually make this vision a reality that not only works for policymakers, but also works for educators and also works for parents and children. Thank you very much.

Yes?

UNIDENTIFIED AUDIENCE MEMBER:  I’m a little bit concerned about some of the changes that might be made at NCLB.  What do you think will happen down the road?

MR.CARROLL:  Yeah I can’t predict with any certainty what will happen.  I think just because the political wind-shift in Washington DC, I’m not sure how this will come out at all. I think there’s a general consensus that the act even among – and I would have counted myself as one of the original enthusiasts for the federal No Child Left Behind Act – I think in terms of the impulse of what it was trying to achieve in terms of literally leaving no child behind, creating a spotlight on those areas where districts and charter schools may not have been up to snuff, I think was very important.

But I think most people would agree that the act micromanages districts and state education departments too much, that the accountability structure for a country this large and this varied is too inflexible. And that, as with almost every grand federal venture, the law of unintended consequences comes into play and so how you sort all of that out with the shifting political power in DC, I couldn’t begin to imagine how this debate will end up.

UNIDENTIFIED AUDIENCE MEMBER: ?Do you anticipate a? higher bureaucracy to audit the accountability?

MR. CARROLL:  Well I have two fears: one is in the suggestion for contracts of excellence, which are actually very similar to the contracts that charter schools enter into with the Board of Regents or the State University.  Having been through that process multiple times, I’m not sure that that’s the most effective use of everybody’s time.

We literally, right now, just for little charter schools, fill out applications that are 500 or 600 pages long. I’m not sure we wouldn’t be better served coming up with something that was just five or six pages. So my fear is that whole process will spawn a bureaucracy, and you have, and I just know, and the schools that I’m involved with are all within five to 10 minutes of the state Education Department. So the prospect of being in a school district that’s hundreds of miles away, and having somebody reading all of these contracts and understanding the nuances on the ground in the district, I find frankly a little frightening.

The second fear I have is, now that the governor has laid this out as a policy issue, that policy makers in the legislature will think that these are all issues that they actually should decide.  And although I don’t always agree with the Board of Regents, I don’t always agree with the state Education Department, and there’s some things I profoundly disagree with them on, I certainly do not think that this state legislature is more equipped to decide a lot of what is going to make this succeed or not, (they) are very technical issues, in terms of do you go with the growth model? A value added model? How do you set up cohorts? How do you measure gains over time? They’re not, whether you have vertical integration of exams?

These are questions that are very important substantively and may lead to the success or failure of the entire accountability system, but for which, at least the legislators I’ve talked to, are not equipped to decide those issues.

So my fear is, on the one hand, a bureaucracy of that one provision of the contracts that he wants to enter into, and the second, that a lot of this stuff should not, I fear, that when it gets into the negotiations in the next few weeks, that legislators, thinking they don’t like certain things that are happening in New York City, they don’t like things happening in other parts of the state, that they will try to take some of these issues and write them into the final legislation.

I think this is going to take years to perfect and even when the education department’s rolled out new exams and? there’s often been mistakes. So I think we need to have it much more flexibly handled by the Board of Regents, and the legislature obviously has oversight of funding and otherwise.

Mr. MCMAHON: We actually do have time for another question or two Tom, but I forgot to make a point coming into this, we’re making a, we’re recording this conference and making a transcript of it, which will be available to you, and so we have a question mic there. So if there’s anybody else who’d like to pursue a couple of questions with Tom, if they would go over to the middle aisle to that microphone, it would be helpful to have and identify yourself, it’ll be helpful for the purposes of our transcript, Thanks.

MR. CAROLL:  Any other questions?  Okay thank you, appreciate it.

Mr. MCMAHON:  Our next speaker, Paul E. Peterson, is the Henry Lee Shattuck professor of government and director of the Program on Education Policy and Governance at Harvard University, and editor in chief of Education Next, a journal of opinion and research on education policy. The author/editor of over 100 articles and 22 books, he received his PhD. from the University of Chicago and was a professor many years there in the Departments of Political Science and Education. He’s also been involved in the Department of Education’s independent review panel to advise the agency in evaluating the No Child Left Behind Act, among his many many other activities in this field.

In introducing him I’ll also note to you, because I know it’s a question that comes up at conferences, we were unable to print out these slides in a format that looked clean. But watch this space, www.empirecenter.org, we will post his PowerPoint in both PowerPoint and PDF format within the next day or two, so you will be able to find it and download it within two days of this conference.

And so with that, Dr. Peterson.

DR. PAUL PETERSON:  Good morning.  I was very pleased to hear that New York has number four as the best because I’m afraid that’s where the Red Sox are going to be in the division this year, so…

Okay, so if we can get this: 10 Accountability Lessons is the theme that I’ve organized my thoughts around. I wanted to begin on a positive note because I’m going to be quite critical about the way we go about accountability so far in the United States, but I want to place that in the context of saying some good things about accountability.

And so my first lesson is overall, accountability seems to have had some positive effects in the states that have introduced it (Peterson, PowerPoint 1, Appendix). So, the next slide (Peterson, PPT 2) we put up, this shows the states which in 1990s, introduced accountability, as compared to the states that did not introduce accountability. And some just reported results and some gave rewards to schools that did well and some did nothing at all.
So a study was done using the National Assessment of Educational Progress, the NAEP, which we’re going to talk about.  I’m sure many of you in the room know what the NAEP is, it’s the nation’s report card and it is the only way we can really monitor what’s going on across the country and compare one state with another.

And this basically looked at what happened to fourth graders four years later, when they were in eighth grade in mathematics on the NAEP. It’s a way of sort of seeing what happened over those four years in states that had accountability and states that did not have accountability and it’s the best study that’s ever tried to get an overall assessment of the impact of accountability.

Now, this is before No Child Left Behind, it was when the states were first beginning to experiment with this. And it shows that those are not massive gains, they’re modest gains of about a percentage point or two. So it’s a plus, it’s not like you’d get excited about it, that this is going to transform the American educational system. But since accountability systems are not that expensive – I’ll get to that in a second – it’s a reform that even in its early beginning phases seemed to have had some positive effects.

The next lesson is on the next slide. Oh wait, a further point on this.  So we can now compare what’s going on in some states with the national average and the two states I thought I would put up there are New York and Florida. And I emphasized Florida because Florida has put together what I think is one of the best accountability systems in the United States and it has introduced a number of other educational reforms as well, so you can’t be sure whether it’s just accountability, but the accountability system, which I’ll be talking about as we go forward here has been a very central part of the reform effort in Florida.
So the Florida bars there (Peterson, PPT 3) are the red bars and the blue ones are the United States as a whole and the yellow one is New York. And this tells you what happened in fourth grade math and fourth grade reading, eighth grade math and eighth grade reading in Florida as compared to what’s happening nationally.

Nationally math scores have been going up at the fourth grade level since over the last decade. But they’ve gone up even more in Florida and they’ve gone up a little bit more in New York, but not done as well as Florida.

So you can see the same thing, not as big a gain in reading, but the same pattern. At eighth grade the gains, the differences are not as large, Florida’s still outpacing things in eighth grade math, at reading nobody’s doing very well.

So that’s the overall pattern.  Let’s go on to the next slide and see what’s happening to African American students (Peterson, PPT 4). Even bigger gains there and, not in eighth grade math, actually New York looks better there. New York doesn’t look good at all in reading, in math, and let’s look at Hispanics.

Hispanics are a difficult group to interpret because there is so much change occurring in that population with the migration coming in, especially in Florida, so New York actually looks pretty good for the Hispanics (Peterson, PPT 5). But whether or not that’s really what’s going on in schools or whether that’s something else that’s happening in terms of migration patterns is hard to disentangle.

So lesson two (Peterson, PPT 6).  Accountability as we know it is not transforming schools. I have sort of already made that point but let me show you in the next slide (Peterson, PPT 7) what I mean by that, which is, by the time students get to graduate from high school, there in mathematics since 1970, their test scores have not really gone up much at all. They went down in 82 then they came back. So they have sort of recovered where they were originally, maybe up a tiny bit.  So that’s the overall pattern.

Now if you look at black students, you see that they made really substantial gains during the 1980s and then in 1990 they tailed off again. And so we didn’t sustain the pattern of closing the test score gap that we had established for 17 year old population.

Let’s look at reading (Peterson, PPT 8) because it’s even more dramatic there. There you see very little gains at all and you see these really remarkable gains by African Americans, then followed by a decline in the reading scores, with a similar pattern by Hispanics.

So the message that you take away from this, when you look at 17 year olds is that whatever was happening at the elementary school level, it’s not being carried through high school. And if you look at graduation rates the same pattern emerges. It’s now evident that we have lower graduation rates in 2000 than we had in 1990. In 1970 we were the world’s leader in high school graduation rates, our high school graduation rates did not improve much after that, and in fact in 1990 they started going down.

The rest of the world, meanwhile, has caught up to us. We are now at the industrial world average instead of being the leading country in the world. Now that is also true of our test scores, by the way, compared to other countries we’re now at the international average, we’re no longer the leading country in the world.

If you want to think about this in terms of the world is flat kind of context, there is just a terrible need to do something about our educational system that we have not been doing over the last decade or two.
My third lesson in accountability (Peterson, PPT 9), lesson three is accountability is cost effective. And I did have a slide that I somehow didn’t make into the packet, which brings out the evidence that if you’re going to have class-size reduction, it’s going to be extremely expensive.

To reduce class size from 24 to 16 is going to increase the cost of education by 50%. And the class size reduction study that everybody’s relying upon, the Tennessee STAR Study, does show that you get gains that are about 20% of the standard deviation, which isn’t bad, but the cost of that intervention is just, the 8% increase is not going to be enough to get the kind of class size reduction that they got in the experiment in Tennessee. It was a lot bigger intervention.

So what makes accountability a very attractive strategy for reform is that if it can actually be done correctly, it’s not a very expensive thing to do. The cost of education per pupil runs around $10,000 in the United States on average, and the cost of testing, introducing a testing program costs about $100 at most on average per pupil in a given year. So it’s a trivial part of a total pattern.

So putting some effort into getting an accountability system done correctly is well worth the cost.

Okay, on to lesson four (Peterson, PPT 10). So the NCLB measuring stick, which is one that’s having a lot of influence in New York, is the one that is terribly flawed. And one way it’s flawed is that the standards vary state by state.

So if we go on to the next slide (Peterson, PPT 11), this is a hard one for you to read. When you can see it in detail, I can’t even see where New York is on there.  Or maybe? [Pause]  So Massachusetts, I want you to say, it’s better than the Boston Red Sox, it’s right up there at the top in terms of our standards. Actually, Massachusetts has introduced some very effective accountability reforms and one of them is that they do set very high standards for students.

Florida is not much better than New York is and most states are at that level. Now this is all comparing it to the NAEP standards, the ones that were set up nationally as the correct way of assessing whether a student really knows what they are supposed to know in fourth grade or eighth grade.

When you compare what the states have done, they have really set much lower standards than those that have been set by the nation. So one of the things that people are talking about with No Child Left Behind reauthorization is shall we have a national standard, where should that national standard be. Because right now Johnny can read in North Carolina but he can’t read in South Carolina. I mean the same performance level will give you a completely different interpretation as to whether or not Johnny can read.

So that’s a huge problem, there’s great variation among the states. Well there’s nothing that New York by itself can do about that, but it is something that is affecting the way people are thinking about No Child Left Behind.

So if we move on to the next slide, (Peterson, PPT 12), one of the biggest problems is that parents are very confused by the NCLB measuring stick because it conflicts with state accountability systems.

Now in Florida is where I’ve looked at this the most closely, where they have a very simple reporting system, A, B, C, D and F. And then, in the case of No Child Left Behind, it’s whether or not you make adequate yearly progress, whatever that means, and whether or not the school is in need of improvement. Well all schools are in need of improvement, so these bureaucratic or legislative compromise phrases are language that has very little meaning out there, in contrast to the very simple A, B, C, D, F system in Florida.

But when you have the two systems running at the same time it can lead to a lot of confusion. Somebody’s actually looked in Georgia. I mean if you have a theory of accountability, it goes sort of like this. A democratic theory of accountability is test scores improve, everybody is happy, school board members get reelected with bigger margins than ever and the community supports its schools.

Now test scores go down, the community’s unhappy, school board members are defeated, they bring in a new school board, they fix it up. That is sort of our sort of democratic theory of accountability.

Well so somebody’s actually looked in South Carolina at whether or not this actually takes place. In the first year it did, there was some evidence that it made life more difficult for school board members if their schools weren’t performing well.

But after the first time out the door, these test scores have had no effect because the news media has been confused as to what they mean and there’s conflicts between the federal standards and the state standards, and the definition of whether it’s a good school or a bad school. And so people are so confused.

Somebody’s also done a study of this in Louisiana has found, even before the hurricane, that you’ve got the same confusion so that accountability system had no effect on how people chose candidates in local elections. We haven’t done that in New York so far but I would be surprised if we found anything different.
So let’s go on to the next slide (Peterson, PPT 13). Here is the way it works actually in Florida with No Child Left Behind. In the year we did this study, which was a couple of years ago, three quarters of the state, schools in the state of Florida were said to be not making adequate yearly progress and only one quarter was making adequate yearly progress. Meanwhile, the state of Florida said that these are it’s A schools and these are its B schools and these are its C schools.

So three quarters or, no, yeah, three quarters or more of the schools in the state were said to get A, B, or C. Over a majority were getting an A and a B and those that were doing very poorly? Oh I know, these are the As here. These are the As that were making adequate yearly progress and these are the As that were not making adequate yearly progress.

You had a lot, about half of the A schools were not making adequate yearly progress. So which do you believe? Do you believe that your school got an A or do you believe that your school isn’t making adequate yearly progress?  Well the two accountability systems are giving you completely different signals.

So one thing No Child Left Behind has to figure out is a way to get its accountability system and the state accountability system on the same page. That’s an issue that New York should try to work out as it moves forward, is to see whether or not you can get the Department of Education, which isn’t necessarily highly responsive to state officials when they go and talk to them about this problem. But it is something that they really need to address as they revise the law and as New York begins to move its plan forward.

So if we go on to the next slide (Peterson, PPT 14), well NCLB, don’t do what NCLB does because they do a terrible job of identifying good schools, the next slide will show that.
This is, here is what we did.  We actually have data from Florida that allows us to tell whether or not kids are learning in that school. Because we know how well the student did last year and how well the student did at the end of this year. So we can compare last year’s performance with this year’s performance to see how much the child learned over the course of that year.

Well my definition of a good school is one where kids are learning. They are learning a lot at a good school, a little, okay, and not at all, bad. Okay so this would seem to me to be a very straightforward way of defining what a good school is.

It’s sometimes called the value added model, it’s sometimes called the growth model, there’s variations on this but the concept itself is very, very simple. How much is a particular child learning in a school in a given year, and what’s the average for all the kids in that school and what’s the average for the lowest performing kids in that school.

Well if you were to take NCLB in Florida (Peterson, PPT 15), their definition of whether or not that school is making adequate yearly progress, and you then decided to send your kid to that school because it was, and not to another school because it wasn’t, you would be making a mistake 30% of the time, 30% of the time you would be making a mistake because the school that you decided not to send your child to was actually doing a better job than the one that was defined by No Child Left Behind as making adequate yearly progress.

Well why is this? Because No Child Left Behind measures performance of a school based on whether or not a certain percentage of kids reach a certain proficiency level. Well that reaching of a certain proficiency level is really easy for kids who come from families who are educating them at home and extremely difficult for kids who aren’t getting that same help from the family.
So there are some schools that have to deal with children who have huge learning deficits and other schools are fortunate enough to be working with kids who do not have educational deficits. So the second group gets a total free pass on No Child Left Behind. They can be a lousy school, where kids are not learning much at that school, but they happen to have excellent students arriving on the scene because they come from a family background that’s providing them with the support that they need.

So because of this level measure, this percent proficient measure, which is also being used here in New York, as Tom Carroll just explained to you, it’s misidentifying in Florida, and I’m sure the same thing is happening in New York, although I can’t prove it. It’s mismeasuring 30% of the time, which of two schools is the better school.

Now the Florida system is better. I won’t explain this chart because it takes too long. The Florida system is better but it’s not perfect because half of the Florida score is based on percent proficient and half is on this growth or value added model. I’ve been trying to encourage the people in Florida to move completely to the value added model because then they could really nail which schools are doing the best job of improving student test performance.

Well to do that you’ve got to have the information to do that. And so let’s go on to the next lesson.

For accountability to work, States need to build a database that can track students over time (Peterson, PPT 16). It’s absolutely essential, you will never have a decent accountability system until you put that into place. It’s not easy to put that into place, it took Florida four years to put it into place, it is now a terrific system, and it can still be further improved. I’ve worked with the data, it makes sense, you can organize it, you can look and see what’s happening in specific schools and specific classrooms, you can see how effective every teacher is in Florida.

Using this data we have been able to identify, we don’t identify names, but we are, we have actually looked at this data and we are able to tell you the characteristics of the teachers who are effective and the characteristics of the teachers who are not effective.

The city of New York has begun to put this together so it’s actually being done in parts of New York. And somebody has just shown that whether or not you’re a traditionally certified teacher or Teach for America teacher or an alternatively certified teacher, or not certified at all, doesn’t make any difference, on average in terms of student performance.

So you can learn a lot about your educational policies once you put into place a tracking system that allows you to track how much students are learning from one year to the next.

So the first thing to do is to put together an accountability system that expects this and the second thing to do is to put into place a data collection system. It can be done, it has been done, it’s being done in North Carolina, it’s being done in Texas, and it’s being done in Chicago. It’s being done in Florida, I know Florida the best. It can be done in New York State, I don’t believe New York State doesn’t have the capacity to do what these other States are doing.

Okay so the next lesson is schools respond if accountability contains a penalty (Peterson, PPT 17). Well this is a point that Tom Carroll made as well.  If we go on to the next slide (Peterson, PPT 18), the evidence for that comes from the Florida voucher program, because some schools said if you got an F two years in a row, you would have to go into the voucher program. That no longer applies because of a state court ruling, but the time we did this study we found that once a school got one F, threatened by a voucher system if it was to get a second F, that school improved dramatically relative to the other schools in the state.

We had a way of doing it so we could make that estimation pretty precise and 10% of a standard deviation gain is a pretty big gain, it’s half as big as the class size reduction one and it didn’t cost. The cost of introducing this is minimal.

So what kind of choice system you want to put into place is an interesting question that the state might want to address. But some kind of a choice system for those who are attending failing schools, such as under No Child Left Behind, is a way of motivating schools to pay attention when their scores are very low.

I don’t think you should have a lot of F schools. I think one of the problems with No Child Left Behind is there are so many schools that are said to not be making adequate yearly progress that nobody really knows what to make of all of that.

So a good accountability system should have multiple cutting points so you can identify the very best schools and also the very worst schools and then you should have some intermediate ranks as well.

Okay, moving on. Lesson nine (Peterson, PPT 19) is student accountability is more effective than school accountability. I know I’m running out of time so I’ll run over this point quickly since it’s really not on your agenda. So I think you should take pride in New York that you do have a high school diploma that requires passage of the Regents exam, because I think there’s good evidence that is more effective than probably any other kind of accountability system.
I think one of the things that the governor said that I think needs to be thought about is, yes, adults should be held accountable when students fail, but you shouldn’t let students off the hook either.

I think that teachers will feel a lot better if they know that students are caring about their own education and are taking responsibility for their own education. And I think sometimes our rhetoric is students are widgets, we just pour this stuff into their heads. But they aren’t that way, they’re human beings, they respond to incentives too.

So let’s just look at the evidence on how they do respond to incentives (Peterson, PPT 20). Theodore Sizer, the wonderful educator who created charter schools and who created great public schools and was a great leader in many ways, once said, “The student is the crucial actor. Whether we adults like it or not, he or she decides what has been purveyed.”

Let’s go on to the next slide (Peterson, PPT 21). This is in Chicago. Once they said you’re not going to get into fourth grade unless you do well enough in third grade and this is what happened to test scores in math in Chicago compared to what would have happened had that not been put into place.

Let’s go on to the next slide.  This (Peterson, PPT 22) is in reading the same pattern. Let’s go on to the next slide. This (Peterson, PPT 23) is comparing what happened in Chicago after that in other mid western cities that didn’t introduce that same plan.

Let’s go on to the next slide. This is from Florida. They did the same thing in Florida (Peterson, PPT 24), put into place you’re going to be held back if you don’t do well on this and this is what happened to the test scores. This is in the state’s own test and this is in the Stanford 9, a national norm test. The jump in reading wasn’t as great as the jump in math, but that happened from one year to the next, once you change the expectation.

Let’s go on to the next one (Peterson, PPT 25), which is sort of interesting because these are the ones that were held back, and they actually did better than the ones that were not held back. So it wasn’t like it harmed these kids, it was that they really needed another year at that grade level in order to have grasped the material.

Let’s go on to the next one (Peterson, PPT 26). This is an international study that looked at what happens when countries have comprehensive examinations, like the Regents exam here, the old Regents exam, closer to the old Regents exam than the new one. It actually, at the eighth grade level, has a huge positive effect on the test score performance of kids in those countries.

So the one thing that differentiates countries around the world who do the best at the age of 15 or the age of 13 is whether or not, when they are going to graduate from high school they are going to have to take a serious examination, not one of these pencil and paper tests but a serious examination with long questions and serious answers. You got to know your Shakespeare in order to pass the test kind of examination. It has a huge impact on how much effort young people put into their education in adolescence.

So let’s go on to the next one (Peterson, PPT 27). Oh this is from Massachusetts. In Massachusetts we put into place a high school graduation exam in this year and this is what happened to tenth grade test scores. This is what happened to test scores at the middle school level and the elementary school level. They did not improve because they weren’t being held accountable.

And this is the pattern that had taken place before the test and then it shot up at the tenth grade level afterwards. And it sustains, it keeps going up, every year it keeps going up, which is amazing. That’s bringing us down to ’06 there.

I think I have one more lesson at the end here (Peterson, PPT 28). And this is a bold one.

If you get the accountability system right, you can then begin to hold teachers accountable, and principals accountable. You can’t do that with the current accountability system, it would not be fair.

So it is pointless to start holding teachers accountable until you get the measuring stick correct.

Thank you.

MR. MCMAHON: We actually have a decent amount of time for questions and I would only ask that you please go to the microphone in the center of the room here?  but if you could go the mike and identify yourself before your question.

JOEL MARGOLIS: On one of the earliest charts where you were comparing New York and Florida. Could you tell me what the metric was on the ordinate.  There was no indication. Everything else was sigma, on that one there was no… It was like 15 compared to 40, I couldn’t tell what the metric was that you were comparing it to.

MR. MCMAHON: Let’s put that slide up if we can, can we go all the way back to the beginning?

MR. MARGOLIS: What was the score?

DR. PETERSON: This is the average change in the test score on the day.

MR. MARGOLIS: Oh, okay.

[INAUDIBLE QUESTION]
DR. PETERSON: Well, you know it’s best to think about these in standard deviations rather than, because every test is different.  It can be very confusing.  The standard deviation tends to be around 25 to 30, depending on the test.

One standard deviation is the difference between a fourth grader and an eighth grader, so 25% of a standard deviation is a full year of learning at this point in their schooling, and so these are not trivial, these are pretty big.  These are like a half to a full year of learning difference, if you’re in that 15 to 20 range.

FEMALE VOICE 1: Are there standard errors around those numbers?

DR. PETERSON: Oh yeah, but remember these are huge samples so the standard errors are very small.

ALLISON ARMOUR-GARB: You mentioned that some jurisdictions put in high stakes tests and retain students in grade based on their test performance. You showed that it didn’t have a negative effect, that they showed learning gains the next year.  I’m wondering what your sense is of the long term effects on the students and the likelihood that they’ll graduate.

DR. PETERSON: Well, we don’t know because this was only introduced recently. But when the retained students actually do better the next year, then those who were passed on, that is a, you know, optimistic indication. But one doesn’t know what the long-term impact would be. It’s too soon to tell.

UNIDENTIFIED MALE VOICE: Dr. Peterson, the last lesson you gave was the holding the teachers, principals etc. accountable. I think the next panel might talk about this but one of the things the governor here is introducing is this idea of a contract for excellence, where if I remember correctly the statutory language in his proposal says that henceforth superintendent contracts must have a provision by wish the commissioner of education under various consequences can remove a superintendent.

I mean is that the kind of, that seems very bold to me. Is that that the kind of thing you’re talking about? I mean is that ultimately the way because these are all dictated by labor contracts right now, so unless that’s negotiated in either at the district level or mandated at the state level going forward?

I was wondering if you could say something about that and whether any other state has things like that.

DR. PETERSON: Well I don’t know exactly what the governor has proposed or how it’s going to evolve as people begin to discuss it.

But one thing that is clear to me is that holding schools accountable is holding an ‘it’ accountable. You can only hold people accountable. I don’t quite understand what it means to hold a school accountable because the only way you can hold a school accountable is to shut it down.

Well how many schools are we going to shut down? You just can’t shut a school down. This is a physical facility. Now you can restructure it. No Child Left Behind says we’re going to restructure these schools but how many schools can you restructure? That’s an extraordinarily difficult thing to do and nobody really quite knows how to do it. And right now, the way this law’s being interpreted, when they do restructure schools they’re doing hardly anything at all in most of the cases.

As best we can tell that’s the evidence we have so far on that. So holding schools accountable is just a very odd thing. But holding teachers accountable in the sense that you know how much learning is happening in a particular classroom, and let’s say three years running we find out this child, the children that take from this teacher is learning a lot. These kids are learning one year more than kids taking classes from other teachers, which is actually happening.

In New York City today, if you take a class from the top 25% of teachers in New York City, they learn one year more in a given year than those who take teachers from the bottom 25%. One year’s difference.

Now if you went to those, that first group and said, the kids are really learning from you and they’re doing this year after year, we’re going to give you a raise. If you went to those at the bottom and said you need to re-think what you’re doing and you worked with them and you asked them to re-think what their doing and if they improve, that’s great, and if they don’t, then you really need to take a look at whether or not they belong in the educational business or whether they shouldn’t go into some other business as well.

Now that seems to me to be a vastly more sensible accountability system than trying to hold schools accountable.

ASSEMBLYMAN JOEL MILLER: I think, you know, we’ve been talking about a lot of things but the actual specifics have been totally left out. Not your specifics, you’ve been excellent, but this whole thing about what the governor has proposed.

A school district is only going to be held liable if they get over a 10% increase in funding. Very few school districts in the entire state meet that requirement. And then it’s four years before a superintendent could be replaced and six years before the school board could be replaced and it’s not going to be the same people.

I mean everything you’re saying is absolutely so. But if you look in New York City with the union structure, the better teachers are in the better schools, the least prepared teachers, the least experienced teachers are put into the worst schools. So even there the statistics are somewhat skewed.

And if in fact we don’t take into account just what you mentioned before, that kids with certain backgrounds have a greater potential to learn than kids without that same background, it becomes very difficult to hold the teacher accountable if in fact the teacher is teaching in a school where kids are coming in where there is absolutely no preparation at home.

So I think everything you say about measuring is absolutely correct. Everything you say about assessing and demanding results is very important. But the actual statistics that go into defining each body and each group and how you are going to hold who accountable for what isn’t nearly as clear when you look at just what the governor really proposed and how the funding works out.

Most school districts are going to get 3%, some of the big cities are going to get less than 10%. Very few schools under this contract are going to be held accountable at all and there’s not that much money, $7 billion over four years or over five years, where just slightly less than $3 billion was already cost driven aid. We’ve raised school funding to the same amount before Governor Spitzer got there with this plan.

So, you know, I think there’s a lot that we have to do on how we educate and structure that before we start firing teachers and firing superintendents. It may sound good but we may have more firings than education.

DR. PETERSON: I take that more as a comment than as a question. I do agree that we have to get the measuring stick right first. If you’re trying to get people to change their behavior and you’re measuring what they’re doing incorrectly, you’re going to get results you don’t want.

MS. LEE-ANN MERTZLAFFT: In the capital district Options with Learning provides teacher training and we service directly, including myself, the children who have either been forgotten or believe they cannot learn to read.

Really I think that what it comes down to is what we need to get to the core of is that teacher. And we need to inspire that teacher to believe that every child can learn. And when I do teacher trainings and when we service these children, the first thing I notice is that there’s that big belief that they just can’t teach that child, that the child just cannot learn.

So there’s need to be an inspiration, an incentive to inspire that those children can learn, and that teachers who make the biggest gains in whatever school you are in – whether you are in the best school or the least school – are the teachers who believe that every child can learn. It’s that child that proves success, not the funding but it’s that one individual.

DR. PETERSON: Much truth in that.

MR. MCMAHON: I think that concludes our time for questioning.  Thank you very much Dr. Peterson.