Faith Based Initiative Works, or Does It?

[Jim Raynor]

Read about it here

Originally Posted by Mark A.R. Kleiman

The White House, the Wall Street Journal, and Christian conservatives have been crowing since June over news that President George W. Bush's favorite faith-based initiative is a smashing success.

When he was governor of Texas, Bush invited Charles Colson's Prison Fellowship to start InnerChange Freedom Initiative, a fundamentalist prison-within-a-prison where inmates undergo vigorous evangelizing, prayer sessions, and intensive counseling. Now comes a study from the University of Pennsylvania's Center for Research on Religion and Urban Civil Society reporting that InnerChange graduates have been rearrested and reimprisoned at dramatically lower rates than a matched control group.

For those who know how hard it is to reduce recidivism, the reported results were impressive. Colson celebrated the report by visiting the White House for a photo op with the president. House Majority Leader Tom DeLay issued a triumphal press release. The Journal smacked critics of faith-based programs for "turning a blind eye to science" by opposing InnerChange. The report heartened officials in the four states that have InnerChange programs and buttressed President Bush's plan to introduce the Christian program in federal prisons.

You don't have to believe in faith-healing to think that an intensive 16-month program, with post-release follow-up, run by deeply caring people might be the occasion for some inmates to turn their lives around. The report seemed to present liberal secularists with an unpleasant choice: Would you rather have people "saved" by Colson, or would you rather have them commit more crimes and go back to prison?

But when you look carefully at the Penn study, it's clear that the program didn't work. The InnerChange participants did somewhat worse than the controls: They were slightly more likely to be rearrested and noticeably more likely (24 percent versus 20 percent) to be reimprisoned. If faith is, as Paul told the Hebrews, the evidence of things not seen, then InnerChange is an opportunity to cultivate faith; we certainly haven't seen any results.

So, how did the Penn study get perverted into evidence that InnerChange worked? Through one of the oldest tricks in the book, one almost guaranteed to make a success of any program: counting the winners and ignoring the losers. The technical term for this in statistics is "selection bias"; program managers know it as "creaming." Harvard public policy professor Anne Piehl, who reviewed the study before it was published, calls this instance of it "cooking the books."

Here's how the study got adulterated.

InnerChange started with 177 volunteer prisoners but only 75 of them "graduated." Graduation involved sticking with the program, not only in prison but after release. No one counted as a graduate, for example, unless he got a job. Naturally, the graduates did better than the control group. Anything that selects out from a group of ex-inmates those who hold jobs is going to look like a miracle cure, because getting a job is among the very best predictors of staying out of trouble. And inmates who stick with a demanding program of self-improvement through 16 months probably have more inner resources, and a stronger determination to turn their lives around, than the average inmate.

The InnerChange cheerleaders simply ignored the other 102 participants who dropped out, were kicked out, or got early parole and didn't finish. Naturally, the non-graduates did worse than the control group. If you select out the winners, you leave mostly losers.

Overall, the 177 entrants did a little bit worse than the controls. That result ought to discourage InnerChange's advocates, but it doesn't because they have just ignored the failure of the failures and focused on the success of the successes.

The Penn study doesn't conceal the actual poor outcome: All the facts reported above come straight from that report. But the study goes out of its way to put a happy face on the sad results, leading with the graduates-only figures before getting to the grim facts. Apparently, the Prison Fellowship press office simply wrote a press release off the spin, and the White House worked off the press release. Probably no one was actually lying; they were just believing, and repeating as fact, what they wanted to believe. It's hard to know for sure what those involved were thinking: Study author Byron Johnson canceled a scheduled interview at the last moment. The White House didn't respond to requests for comment.

InnerChange program manager Jerry Wilger says he doesn't know much about research, but he doesn't think it's fair to count the performance of the people who dropped out of his program against him, a fair-sounding objection that misses the point entirely. If InnerChange's 177 entrants were truly matched to the control group but ended up having more recidivism, then either the apparent success with the graduates was due to "creaming" or the program somehow managed to make its dropouts worse than they were to start with. If the program genuinely helped its graduates and didn't harm its dropouts, and if the whole group of entrants was truly matched to the controls, then the group of 177 should have done better than the controls. And they didn't.

So, the feel-good winners-only analysis simply isn't worth the paper it's printed on. Only the full-group analysis (known technically as "intent-to-treat," a holdover term from its origins in medical research) has any real value. And on that analysis, the program has a net effect of zero or a little worse than zero. That makes it a loser.

John DiIulio, an intellectually serious advocate of faith-based programs who was the first director of the Bush administration's faith-based initiatives and the founder of the Penn research center, acknowledges frankly the results weren't what a supporter of such programs would have hoped for. But he points out that a single study almost never provides a convincing yes or no answer on a program concept. "The orthodox believers point to a single positive result and say it proves faith-based programs always work. The orthodox secularists point to a single negative result and say it proves faith-based programs never work. They're both wrong."

The poor result of InnerChange doesn't mean that no faith-based prison program could work, but it does mean that this one hasn't, at least not yet. It joins a long line of what seemed like good ideas for reducing recidivism that didn't pan out when subjected to a rigorous evaluation. Maybe my own pet, literacy training, wouldn't do any better in a real random-assignment trial. But that's why you do evaluations; they tell you things you didn't want to hear. If you're honest, you listen to them.

And if you're smart, you don't listen the political advocates of "faith-based" this and that when they say they're only asking us to support programs that have been "proven" to work.

[Darth Wong]

Fundies are very good at manipulating statistics. The various studies used to "prove" that faith-based healing works use similar tricks.

[Gil Hamilton]

Fundies are very good at manipulating statistics. The various studies used to "prove" that faith-based healing works use similar tricks.

Whoa whoa whoa! We all know that faith based healing works because of the 22 people who got better in the Faith Based healing group, a whole 89% stayed better. This is compared to a paltry 64% of the people in the 189 person control group getting better, which used "conventional" (IE Satanic) medicine. How dare you question faith based healing! I demand our tax dollars go to faith based programs!

[darthdavid]

"The orthodox believers point to a single positive result and say it proves faith-based programs always work. The orthodox secularists point to a single negative result and say it proves faith-based programs never work. They're both wrong."


That's a classic.

[NapoleonGH]

wow big surprise attempting some form of rehabilitiation in prisons works...maybe the lessen has nothing to do with faith based programs and has something to do with rehabilitation actually being a good idea (as japan shows, they have something like a 1% re-arrest/re-conviction rate, basically when you leave a japanese jail you leave having been thoughouly trained in useful job, and it appears that it works)

[Admiral Johnason]

I think I need to start a demination to get rid of fundies. They are the retarded little kids of the Christian church. They make all the faithful look bad.

[Darth Wong]

I think I need to start a demination to get rid of fundies. They are the retarded little kids of the Christian church. They make all the faithful look bad.

Actually, they are the only ones who take the written tenets of the faith to their logical outcome, no matter how insane that outcome may be. The problem lies in the Bible itself, and even the moderates have serious trouble with the idea of improving the Bible.

In other words, moderates may differ from fundies, but they create and sustain the environment necessary for fundies to flourish.

[The Duchess of Zeon]

Actually, they are the only ones who take the written tenets of the faith to their logical outcome, no matter how insane that outcome may be. The problem lies in the Bible itself, and even the moderates have serious trouble with the idea of improving the Bible.

In other words, moderates may differ from fundies, but they create and sustain the environment necessary for fundies to flourish.

That's the elegance of Catholic doctrine. The idea of an intercessory who can interpet the Bible and add to it a living canon of the faith is brilliant; it lets your religion adapt to changing times by having your infallible heir to the Apostle Peter, of course, change the doctrine as necessary. There are, of course, downsides to the Catholic system, some of which have notoriously come up of late--and, of course, the fact that the current Pope has refused to change to fit the times. It will take his successor, perhaps.

In the long term I suspect that Catholicism will prove much more lasting than any Protestant denomination. The advantage of the Protestants, however, is you can simply create an entirely new denomination and interpet the Bible as it suits that denomination--but individual denominations that are excessively conservative will die out as society changes around them.

[Howedar]

Actually, they are the only ones who take the written tenets of the faith to their logical outcome, no matter how insane that outcome may be.

Absolute literal interpretation of a document that has been translated through several different languages is not logical.

[Darth Wong]

Actually, they are the only ones who take the written tenets of the faith to their logical outcome, no matter how insane that outcome may be.

Absolute literal interpretation of a document that has been translated through several different languages is not logical.

The smarter fundies take that into account, and look closely at the various translations. Some of them even learn to translate the original languages.

The point remains: a religion which bases its doctrines upon the authority of a document can scarcely point fingers and laugh at people who take that precise concept and carry it to its logical (albeit deranged) conclusion. You are speaking only of flaws in implementation.

[Admiral Johnason]

Actually, they are the only ones who take the written tenets of the faith to their logical outcome, no matter how insane that outcome may be. The problem lies in the Bible itself, and even the moderates have serious trouble with the idea of improving the Bible.

In other words, moderates may differ from fundies, but they create and sustain the environment necessary for fundies to flourish.

That's the elegance of Catholic doctrine. The idea of an intercessory who can interpet the Bible and add to it a living canon of the faith is brilliant; it lets your religion adapt to changing times by having your infallible heir to the Apostle Peter, of course, change the doctrine as necessary. There are, of course, downsides to the Catholic system, some of which have notoriously come up of late--and, of course, the fact that the current Pope has refused to change to fit the times. It will take his successor, perhaps.

In the long term I suspect that Catholicism will prove much more lasting than any Protestant denomination. The advantage of the Protestants, however, is you can simply create an entirely new denomination and interpet the Bible as it suits that denomination--but individual denominations that are excessively conservative will die out as society changes around them.

Just for the record, I am only Protestant because I actually do have some beef the Catholism. When they change, I'll join.

[The Dark]

Meh. Theologians were saying back in the 100s CE that the Bible couldn't be taken literally by anyone with any sort of intelligence. If it held true almost 2000 years ago, it certainly does now.

I was reading the book Stealing Jesus by Bruce Bower a few weeks ago. He's a homosexual Episcopalian (non-clergy) who was writing on how the fundamentalists have perverted Christian ideals. From his view, the main difference between the two lies in the central scripture they follow. The fundies follow the Great Commission: "Go and make disciples of all men." The moderates/liberals/traditionalists, on the other hand, follow the Great Commandment: "Love the Lord thy God with all your heart, mind, and spirit, and love thy neighbor as thyself." This is the fundamental difference (pardon the pun) between the two groups, and the main reason it's relatively unknown is because there really are no high-profile moderates, let alone liberals, from the Christian community. All the clergy the media deals with are conservative to fundamentalist.

[Darth Wong]

Meh. Theologians were saying back in the 100s CE that the Bible couldn't be taken literally by anyone with any sort of intelligence. If it held true almost 2000 years ago, it certainly does now.

Of course it does. But once you accept the doctrine that Biblical authority can override direct observation and logic on any level, then the fact that the Bible doesn't make sense becomes irrelevant; for the fundie, it is true anyway.

That's the problem: the doctrine of Biblical authority as supreme, which the fundies merely take to heart. Not all Christians accept this doctrine, but it's not exactly a small minority either.