by: ThinkOfTheChildren 08 April 2013 09:29PM, first published on Less Wrong

I recently saw this post and wondered what the thoughts of The Tutor Report reader would be. Please let me know in the comments section.

[divider top=”0″]

I work for a consulting firm in the US which advises the local, county, and state government with regards to education policy. I will briefly outline how the education system in the US works, as it’s probable most readers are unaware. Most of the money/resources schools receive comes in the form of grants. Some institution, philanthropist, or government, notices a particular problem in the school system, say, the achievement gap between African-Americans (hereafter, blacks) and Caucasians (whites). A school (or school system, consulting firm, organization, church, or individual) comes up with a strategy for fixing the problem; this is called the grant proposal. The institution (or whatever) offering the grant selects the proposal (or multiple proposals) which it deems worth pursuing, and gives the school (or whatever) the resources outlined in the grant. Usually, after the strategy has been enacted, the grant requires that an outside evaluator be hired (usually with a portion of the grant money) to evaluate whether the strategy was successful, and this influences whether the school will get the grant again next year. My firm is such an outside evaluator, although we serve other functions.

However, this process yields sheer lunacy, mostly because of the ridiculous ineptitude of every single person involved. I remember specifically the first grant project I helped to evaluate. The local state government was offering up to $2,000,000 for grant proposals which would help the students in grades 6-8 who had failed their end-of-year standardized reading exam (a well-made test, in my opinion, in which failure basically means illiteracy). The specific project I was evaluating had only gotten $800,000 out of the maximum $2m. Its strategy was to purchase the male students iPod Touches, the female students makeovers, manicures, and pedicures at a local beauty parlor, and all students were offered an additional iPod Touch or Makeover, respectively, if they passed the exam at the end of the current year. The grant proposal had specifically listed these actions as being the goal of the proposal. If the iPods and makeovers were purchased, that constituted success.

When I asked the man who was in charge of the project if he really meant that these actions were the ‘strategy’, not the ‘goal’. He expressed confusion; he thought if the male students had iPod Touches, they obviously would get better at reading, and if the girls got makeovers, it would improve their self esteem and they would be more confident and get better at reading, so obviously isn’t the goal of the project to purchase iPods and Makeovers for the students? I explained to him that the goal was to make students, who had previously failed the exam, pass it on their next try. Success would, obviously, be measured in terms of how many students passed the exam. The strategy was whatever actions you took, whereas the goal was what you were trying to achieve. Now that the project was over, I told him that he had to go look at the reading scores and see if they improved. He couldn’t understand why he had needed to do this, and indeed, refused. I asked him how he had identified the students who he needed to give iPods to in the first place. Did he use their reading scores? Did he ask the school for a list of students who had failed the exam? No. He had asked the school for the free-lunch list, which determined which students came from low-income families, and for the bus route list, which determined which students came from low-income areas. He picked out any students who were on both of these lists who were also black. Since black students tend to have low reading scores, and low-income students tend to have low reading scores, those are the students who need the most help, and so are the students he targeted with his project. When I asked the school for the list of students who had failed the reading exam, it turns out that only 25% (14/56) of the students targeted by the program had failed the reading exam in the first place.

When I wrote up my evaluation, I described in rigorous detail everything the man had done wrong, put in a strong recommendation to not award him grant money in the future, and suggested that some sort of corruption investigation be conducted to see if he had committed any crimes (23 iPods + 23 Makeovers does not total to $800,000, after all). When I submitted this to my boss for approval, she was flabbergasted, and explained that the evaluators job was to collude with the grant proposal submitter, so that we got more evaluation jobs from them in the future. Over the next couple days, we had a long conversation, and in the end, she allowed my evaluation to go through.

The next project I evaluated was just as criminally neglectful as my first. And the next. And the next. In fact, for the first three years I worked at the firm, every single project I evaluated listed their ‘process’ and then said that their ‘goal’ was to enact the process. Every single project had used any subsidized lunch lists, bus route data, or demographic data they could get their hands on to decide which students to target; not a single project actually looked at test scores, for deciding either which students to target or figuring out if the project had even succeeded.

After three years, a large think-tank associated with dropout prevention noticed my firm’s scathing evaluations (with some admiration, I think), and asked us to give a presentation to all school counselors state-wide on how to properly design a grant proposal. We included such advice as, “if your project is supposed to help students who failed Algebra, you should target students who failed Algebra, not the demographic groups who are “known” to fail Algebra”, and “if you want to help students who can’t read by the third grade, and you come up with a good idea for how to help them, your project’s success is measured by looking at the kids who you helped and seeing if they improved their ability to read. Success isn’t determined just by successfully implementing your idea.”

We gave that presentation, and then walked around and helped the counselors with their current grant proposals. I personally walked around and talked to at least 50 individual teams working on grant proposals. I’d say maybe a third understood what we had said, and were able to write their own grant proposals after that such that they knew what constituted success or failure, and made sure they were targeting the right students. A third. This is, of course, absolutely unacceptable.

Unfortunately, soon after we gave that presentation, the situation became political. The problem with having teachers look at the actual, real data, is that it very soon becomes abundantly clear that the school system is routinely f*****g over certain demographics of students. Just as the process of picking students for your learn-to-read project has to do with whether they’re black and how much money they make (and nothing to do with whether they actually know how to read), it turns out the process of deciding who gets to take advanced classes, as opposed to the ridiculously boring remedial classes, is also based almost entirely on the free lunch lists, bus route information, and other demographic data. While such data is correlated with success somewhat (that is, poor/black students are slightly less likely to succeed than rich/white students, all else being equal, for whatever reason), obviously it’s still better to use actual test scores to decide, rather than something you’re using to predict the test scores. It’s a bayesian base rate confusion thing: if there are 10,000 whites and 3,000 blacks, and 25% of the whites need help, and 30% (or 26%, or 25.1%, or however small of a difference you need there to be before you are willing to admit it) of the blacks need help, then targeting the black kids for help will cause you to, A), miss out on the 2,500 whites who need help and, B), ruin the scholastic careers of the 2,100 blacks who you shoved into the remedial classes who didn’t need to be there.

Can you see where the politics might come in? There were rich whites who got upset when the data told the schools to put some blacks in their son’s advanced math class. There were math teachers who absolutely refused to allow blacks/poors in their classroom, or worse, treated them in such a way as to cause them to fail, thus confirming their worldview. Luckily, my firm started collecting data on teacher aptitude some time ago, and basically you can separate all advanced math teachers easily into two categories:

Okay with blacks in their classroom. Blacks and whites both end up succeeding at equal rates. Not okay with blacks in their classroom. Whites end up succeeding, blacks end up failing.

It’s pretty hard to argue with the data. Anyway, the problem is, the total number of high-level math teachers is in very, very short supply. Usually in each district, the parents of the white children are more willing to advocate for their placing into advanced classes, and so the whites get those teachers while the blacks don’t. Currently, my firm and its allies are trying to push the government into forcing the schools to use a Bayesian prediction model, in which you feed an individual student’s test scores for the past 5 years and it spits out their probability of success in the advanced classes, and you keep putting the students with the highest probability of success in the top classes until you run out of teachers.

We’ve convinced a few individual schools to use this system. A certain principle plugged in all of the data he had on his students, checked some boxes on the program, and got a list of 77 (black, poor) students who were currently about to go into the lower math class, who were predicted as succeeding in the upper math class with >99% probability. After some deliberation and assurances from my firm that we would accept the political fallout if it all went to hell, he put those 77 children into 8th grade algebra instead of 8th grade ‘tech math’. Every single one of them passed, most got Bs or As. It was the most (and, though he wouldn’t say it except privately, only) successful project he had ever seen, and remember that as principle of a school, he probably is aware of between 8 and 12 projects going on at his school at any given time.

So, the real obstacles are pretty obvious: get more good math teachers because we don’t have enough for all of the students capable of learning higher math. Get whoever is in charge of math placement in your area to make sure the criteria for math placement is based on achievement data, as opposed to demographic data. Get the grant funders to understand the difference between ‘strategy’ and ‘goal’, so that they don’t issue money to lunatics. Get the grant proposal writers to understand that difference, so that they don’t criminally waste taxpayer money. But most of all, the most pervasive problem has been that most people in charge know that there isn’t actually any way to fix the problems, and they just want to do something so they look like they’re helping.

Anyway, this is an important problem that I’m working on, but literally I can only make a difference for my county in my state, and it’s clear that the problem is everywhere. If you are a decision maker in education in your area, please, please, please look into the various Bayesian predictive models used for math placement; they are well worth the money (compared to what your school system is already spending money on) and can make immediate, huge improvements for your school system, which even the politicians should be in favor of. If you are a parent, please advocate for your child to be placed in the upper track. Get test scores, get previous grades, get everything you can. 8th grade math placement is the #1 indicator of success (higher income and later life opportunities), and the people in charge at your child’s school have no idea what they are doing in terms of your child’s actual aptitude and whether they can be successful in the advanced classes.

Regardless of who you are, spend five minutes investigating what your school district’s Math Placement Criteria is, and whether that criteria makes sense to you. You’re likely to find “teacher recommendations” stated as being valuable for the criteria: this is a red flag that, if you intend to raise a child in this district, you will need to intervene regularly in their decision making process, because the evidence shows that teacher recommendations have zero correlation with aptitude in a field. And if you could make my job easier by getting involved in actively solving this problem in your area, then later when the firm I work for goes national, we’ll be sure to thank you.

Please, think of the children!

Orignal Article with Comments may be found here

[divider top=”0″]

oaSES Online, student tracking software with staff, billing, payroll and reports. Request a personal online demonstration on how oaSES can revolutionize your back office procedures.