In Part 1 we talked about the reasons the authors felt some of the existing research was not credible. They wrapped up their critique of the Kanin study with the following quote (and if you read Part 1, you can probably guess who they are going to cite):

As a result of these and other serious problems with the “research,” Kanin’s (1994) article can be considered “a provocative opinion piece, but it is not a scientific study of the issue of false reporting of rape. It certainly should never be used to assert a scientific foundation for the frequency of false allegations” (Lisak, 2007, p. 1). In contrast, when more methodologically rigorous research has been conducted, estimates for the percentage of false reports begin to converge around 2-8%.

Before I get into the primary study that they reference, let me give you an idea of just what kind of games they have in store for us. Here is what they say about the second study used to back up their 2-8% range:

For example, Clark and Lewis (1977) examined case files for all 116 rapes investigated by the Toronto Metropolitan Police Department in 1970. As a result, they concluded that seven cases (6%) involved false reports made by victims.

Ok, so we are back to using data from a single police department, in a single year, and in this case from about 40 years prior to when the article was written, but at least it is in the range they are claiming. They then go on to say this

There were also five other reports made by someone other than the victim that were deemed by these researchers to be false (e.g., a relative or boyfriend).

So, to clarify, the study found that 7+5 = 12 out of 116 reported rapes were false. For most people that would be a false reporting rate of 10.3%, which is outside the 2-8% range , but apparently not for the authors. It would seem that rape reports made by someone other than the victim do not count as false despite being, well, false . At this point all I can say is that at least they are quoting the research of others – if this is the kind of logic they are using, I can only imagine what the results would be if they got to design the study themselves.

Back to their other sources – what is the primary study that they cite?

For example, in a multi-site study of eight U.S. communities involved in the “Making a Difference” (or “MAD”) Project, data were collected by law enforcement agencies for all sexual assault reports received in an 18- 24 month period. Of the 2,059 cases that were included in the study, 140 (7%) were classified as false.

A multi-site, multi-year study? Seems like we actually have some good data to work with here. They then give a website where you can find out more information about the MAD Project, http://www.evawintl.org. Clicking through you’ll find out that their primary source for a “methodologically rigorous” study on the number of false rape reports comes from the organization End Violence Against Women International. If someone tried to present their view that the number of people killed by cigarettes each year is actually pretty low and their primary source was a study done by Phillip Morris, how much credence would you lend that research? What about a study on gun violence done by the NRA? And yet, for some unknown reason, the authors don’t seem to think that End Violence Against Women International might in some way be biased as to frequency of false rape reports.

As it turns out, there is actually a very good reason why the authors don’t see a problem with that, which they are pretty upfront about – and by “are pretty upfront about” I mean “bury it in the footnotes.”

1 Dr. Lonsway is the Research Direcotr [sic] of EVAW International; Sgt. Archambault is the Executive Director of EVAW Internatiaonal;… This article is an adapted excerpt from the training module of the same name in the On-Line Training institute hosted by End Violence Against Women (EVAW) International, at http://www.evawintl.org/evaw_courseware.

So not only do they try to disguise the fact that the main source for their hypothesis is a study that two of the authors conducted themselves for an organization with a clear stake in the matter, but the entire article is essentially an advertisement for their training sessions. Think I’m exaggerating? Check out the last page of article:

(click to enlarge)

Perhaps I have not been keeping up with the times, but I hadn’t realized that scholarly articles came with coupons now. And we are supposed to believe that Kanin’s study is the “provocative opinion piece”?

So how did they get to the 7% number? The first problem we are going to run into has to do with the research itself. Take a look at EVAW’s page on the MAD Project. Despite the data collection period being from 2005-2006, there is very little in the way of actual published research. They have a link to research materials where you can see some of the data collection forms they used, a pdf on the research methods, and a collection of slides listing some of the basic data results, but that is pretty much it. That basic data results document is over 5 years old at this point, it has been over 8 years since the data collection periods, and there is still no comprehensive study document available, which strikes me as a little strange.

One of the key questions in evaluating their claim is going to be “How, exactly, is a false report defined?” As it turns out, that part isn’t entirely clear cut. In fact, it isn’t until paragraph 6 of the section entitled “What is the Actual Definition of a False Report?” that we get to this:

Rather, investigators and prosecutors must base all final judgments of a sexual assault report on the findings from a thorough, evidence-based investigation. The determination that a report is false can then only be made when there is sufficient evidence to establish that the sexual assault did not happen (was not completed or attempted.)

OK, so not a terribly unreasonable definition, but to understand how this has the potential to skew the data, we have to go back to the research methods I mentioned before. If you read through it, you’ll find that the 8 sites were not chosen randomly. Communities had to apply to be in the study and there were 88 US applications in total. These applications were then evaluated by a selection committee to select the final 8 cities . After selection, the 8 communities came together for a 3 days training conference, and that is where I think things get particularly interesting. While they don’t specify the exact training that occured, we know that this article is adapted from some of their training materials. So what kind of things does their training have to say about false rape reports?

To start off with, here are some things that someone filing a rape report might do, which the authors say happens for perfectly justifiable reasons

“alter or exaggerate the details of what happened is to create a case that seems more believable”

“report that the suspect used a weapon when this is not really true”

“describe threats of physical violence that were not really made”

And if that happens?

When we think about these dynamics, it makes sense why victims might provide inconsistent, incomplete, or even untrue statements. Yet many investigators and prosecutors have seen this as evidence of a “false report.” In fact, none of these situations meets the actual criteria for a false report

Um, OK. Doesn’t that create credibility problems for the person making the report though? I mean, if there are valid reasons why a person might report things that didn’t happen, how can we believe the rest of what they report is accurate?

For all of the reasons provided above, it is understandable that victims often give information in their statement that is incomplete, inconsistent or even untrue. Nonetheless, these issues can destroy the victim’s credibility if they are not handled by criminal justice professionals.

“handled”? Uh oh, that sounds kind of ominous. At first the authors just describe exploring issues “gently and nonjudgementally.” A little later on though, they get to this suggestion:

Problems such as inconsistent statements from the victim can also be decreased by reducing the number of unnecessary professional contacts

Are they suggesting that to prevent credibility problems that result when important elements of the accuser’s story turn out to be false, they should make sure they aren’t asked what happened too many times?

This does not mean that investigators and prosecutors should be reluctant to conduct follow-up interviews during the course of the investigation, as additional evidence and information is uncovered. In fact, such follow-up interviews are necessary to conduct a comprehensive investigation.

Phew, I was worried for a second there.

The purpose of any follow-up interviews should therefore be to gather additional information and clarify any questions, not to go over the same information again.

So ask them as many questions as you want, just not about anything you’ve already asked them lest they give a different answer. Gotcha.

However, as important as it is to seek clarification of such inconsistencies or omissions, it is equally important to highlight the accuracy of other details in the victim’s statement. Thus, a primary goal of any sexual assault investigation will be seeking corroboration for details in the victim’s account of events, regardless of whether or not they are relevant for establishing an element of the offense.

She said there was a gun when there wasn’t. She also said he threatened her with violence when he didn’t. Buuuuuuut she said she had a salad for dinner last night and we found a receipt for it, so story checks out. Book ’em.

OK, but what if after all of the gentleness, and not asking the same question more than once, and not getting too worked up about any key elements that turn out to be false, what if after all that the police still feel like something isn’t quite right?

As McDowell and Hibler (1987) describe, any effort to challenge the validity of a sexual assault report could be devastating if the suspicion is misplaced and the victim really was assaulted. Such a challenge would certainly destroy the trusting relationship that must develop between criminal justice professionals and victims for successful investigation and prosecution.

and

Given the size of the caseload that most investigators and prosecutors handle, it seems difficult to justify the inordinate time that would be involved in investigating and prosecuting someone for filing a false report—given that it is typically only a misdemeanor offense.

and

While it is understandable that investigators might want to prove that the report is false out of a sense of frustration and a determination to get to the truth, this is probably not the best use of limited resources.

and

It is also important to keep in mind that most false reports of sexual assault are typically the result of personal and emotional problems, rather than vengeful motives.

I have to detour again here. There is no source cited for that last statement. In the Kanin study from Part 1 of this series, there was an analysis of the motives behind false reporting, but the authors dislike that study so much they pretend it doesn’t exist. You may think I’m being hyperbolic here, but after they stated the results from the MAD study they said the following:

To date, the MAD study is the only research conducted in the U.S. to evaluate the percentage of false reports made to law enforcement.

See? The Kanin study doesn’t exist anymore. This ends up not being a trivial point either, because journalists aren’t always the most thorough. In a fact checking column on the topic of rape statistics, the Washington Post took them at their word that the MAD study was the only one done in the US.

The “Making a Difference” Project, which used data collected by law enforcement agencies over 18 to 24 months, found 7 percent of cases that were classified as false. That study is the “only research conducted in the U.S. to evaluate the percentage of false reports made to law enforcement,” according to the National Center for the Prosecution of Violence Against Women.

Detour over, back to my point.

Do you see the situation they are setting up here?

A rape report can only be classified as false after a thorough fact-based investigation (a confession that the report was false is not enough)

Don’t ask the same question twice so they can’t change their story

Even if you find key parts of their story to be false, don’t pay much attention to it

Challenging the validity of their report could be devastating

Even if you think the report is false, is it really worth your time to prove that it is?

If you need to do a thorough investigation to classify a report as false, but you can’t ask too many questions, you can’t challenge the validity of their story, and you probably shouldn’t be wasting time trying to investigate a false report, how can a report every be classified as false? While the authors present this 7% false report rate as very low, I’m kind of shocked they were able to classify that many using these procedures.

Finally, we get to what I think is the most deceptive part of their false report statistics. When they say that only 2-8% of rape reports are false, the implication is that the rest are true. Based on the ways their source data was constructed though, “2-8% of rape reports are false” would only be an accurate statement if you include the addendum “at bare minimum.” 2-8% isn’t actually a ceiling, or even a range, it is merely a floor.

Let’s take a look at what the MAD study actually found. On page 9 of the basic results we find the following chart:

Unfounded/false is the 7.1% we’ve already dealt with, but what is Unfounded/baseless? EVAW actually has a helpful document on that. It includes the following description of baseless:

Cases determined to be baseless include those that do not meet the elements of the offense and those that were improperly coded as a sexual assault in the first place.

They even give an example

an adult woman is coerced into having sex when her boyfriend who said he would break up with her if she doesn’t do what he asked.

See? The report is unfounded but it isn’t Unfounded/false, so it doesn’t count. What about Closed as an informational report? If you looked at the Disposition section of the police tracking form you’ll see a fuller description

Closed as an informational report (elements of a sexual assault offense not met)

Once again, there wasn’t actually a rape, but the report doesn’t get counted as false. Those last two categories combined make up 26.4% of the total rape reports. Isn’t manipulating statistics fun?

Suspended/inactivated are the cases that there isn’t enough evidence for an arrest and make up another 28.6% of the cases.

Exceptionally cleared is a bit trickier. EVAW has this document which goes into a bit more detail about clearance methods, and the exceptional clearance section starts on page 5. It is supposed to be for when the police have everything they need to make an arrest, but for some reason out of their control, they cannot (for example, the suspect dies or is arrested in another jurisdiction). As they put it in the article.

It is therefore clear that the purpose of exceptional clearance is for police agencies to “count” cases as cleared when they have done their job, but they were prevented by some outside factor from moving forward with an arrest and prosecution. As described by the FBI, exceptional clearance offers a way for police agencies to clear offenses once “they have exhausted all leads and have done everything possible in order to clear a case” (UCR Handbook, p. 80).

How it is used in practice?

It is clear that this is how the definition was interpreted by personnel in the two police agencies involved in the research; the most common reason given for exceptional clearance was “a prosecutorial declination to file charges because of insufficient evidence” (Spohn & Tellis, 2011, p. 125). Comments from expert reviewers also suggested that this practice is in place in other agencies as well.

This isn’t particularly surprising. There are only so many ways police can dispose of a case. If they can’t make an arrest, and they can’t easily put it as false, baseless, or informational report, the only choices left are suspended/inactive or exceptionally cleared. If you were a police officer, a profession where one of the primary performance metrics is clearance rate, would you say “Hey, we did our job and found the guy, and we’d arrest him for it too, but the darn prosecutor just wont let us” and clear it, or would you put it in suspended/inactive?

Finally, we get to arrested, which makes up a mere 20%.

The best way to demonstrate how misleading this methodology is, is to use their data and hyper-conservative classifications to answer a different question: What percentage of rape reports are true?

The 20% of reports that ended in arrest is a good starting point, but being arrested for a crime is hardly solid proof that you actually committed it. Luckily, there is another chart in the MAD results that can help us out. On page 12 we find the breakdown of Prosecution dispositions. They don’t provide enough information for us to know how many of the cases in the police disposition results are in the prosecution disposition results, but we can use them to make some simplifying assumptions.



From this we can see can see that 39.1% of the cases end in either a guilty verdict or a guilty plea. Multiplying that by the 20% of police cases that result in arrest, we are left with the 7.8% I used at the start of my first post. Even this isn’t a good number to use though. If a confession that a report is false isn’t enough to classify the report as false, then the corollary is that a guilty plea isn’t enough to classify a case as true. After all, it would not be difficut to imagine a scenario where, for any number of reasons, someone pleads guilty to a crime that they didn’t commit. 20% x 5.9% leaves us with a “true” rate of just 1.2%. Even if we decide to be generous and include not just the 20% arrested, but also the 17.9% exceptionally cleared, the number still only goes up to 2.2%

At the end of the day, my point is this – If someone tries to advocate a particular policy based on the fact that “only 2-8% of rape reports are false” an appropriate response might be “Sure, but only 1-8% are true.”

I’ll close with one of the quotes from the article I agree with, and I think it sums up my thoughts on the topic pretty well