A study reported by the New York Times on Monday claimed to find “surprising new evidence” that there is no racial bias in police shootings. But the study, and the New York Times’ reporting, uses a small sample of data that leads to simplistic conclusions.

The author of the study, Roland G Fryer Jr, analyzed 1,332 shootings between 2000 and 2015. However, the way he and a group of student researchers created their data was largely by coding police narratives rather than considering the testimonials of witnesses or suspects (assuming that the suspects were not killed by the police in the shooting). The study therefore assumes police reports are unbiased sources of information about facts like whether or not the officer shoots the suspect before being attacked.

There are other serious weaknesses in the research. To understand lethal use of force, Fryer looked at police reports from just one city: Houston. There, he found that blacks were either less likely to be shot by an officer or there was no difference between blacks and whites. Even if the data from Houston were accurate, it is doubtful the city is representative of the country.

The Houston police department also allowed the researchers to look at “interactions with police where lethal force may have been justified”. But that data assumes that all officers make fair and objective decisions about the lethal use of force – decisions that are not affected by the race of the suspect.

Looking at just one other city would suggest very different conclusions. In Chicago, a review of the reports of each police-involved shooting looked at fatal and non-fatal shootings. Despite the city being one third black, a disproportionate 118 black males (44 of them fatal) were involved in the 150 shootings recorded since 2010.

Fryer, a professor of economics at Harvard University, also looked at data from New York City where he found blacks stopped by the police were about 17% more likely to experience use of force. However, Fryer assumed shootings are not necessarily linked to a more general use of police force. Such an assumption seems hard to support: a black person in New York who is stopped by the police is 24% more likely to have a gun pointed at them than a white person, so why would they be no less likely to be shot by an officer? The two seem inextricably linked.

Even where the study did find a racial bias in police behavior, it might underestimate scale of the issue. For example, the data suggests that in New York, black suspects who were compliant (did not verbally threaten police officers, were not arrested and were not found with weapons or contraband) were 7% more likely than white suspects to be pushed to the ground. However it is possible that a white citizen acting in the same way as a black citizen is more likely to have their behavior deemed “compliant” by an officer. If so, the dataset used by the study could be flawed.

The study has also not yet been peer-reviewed, a standard practice in academia where other experts will scrutinize an author’s findings to see if they’re accurate enough to publish.

A wider body of research suggests Fryer’s study is not indicative of a wider picture. Examples include: