The supreme court on Monday appeared likely to say states cannot rely on intelligence test scores alone in borderline cases to determine that a death row inmate is mentally able and thus eligible to be executed.

The justices heard arguments in a challenge from a Florida inmate who says there is ample evidence to show he is mentally disabled and protected from being put to death, even though most of his IQ scores have topped 70. That score is the widely accepted as a marker of mental disability, but medical professionals say test results have a margin of error and in any case are just one factor in determining mental disability.

The decisive vote appears to belong to Justice Anthony Kennedy. He repeatedly questioned the state’s argument for a rigid cutoff.

“Your rule prevents us from getting a better understanding of whether the IQ score is accurate or not,” Kennedy said.

The case centres on how authorities determine who is eligible to be put to death, 12 years after the justices prohibited the execution of the mentally disabled.

The court has until now left it to states to set rules for judging who is mentally disabled. In Florida and certain other states, an intelligence test score higher than 70 means an inmate is not mentally disabled, even if other evidence indicates he is. The Florida supreme court has ruled that the state law regarding executions and mental disability has no wiggle room if an inmate tests above 70.

Inmate Freddie Lee Hall, convicted of killing a 21-year-old pregnant woman in 1978, has scored above 70 on most of the IQ tests he has taken since 1968. Other evidence indicates he is mentally disabled.

The focus of questions by Kennedy and the four liberal justices who appeared sympathetic to Hall was on whether he should get the chance to persuade a court that, taking account of all relevant information, he is mentally disabled.

Justice Stephen Breyer said a defendant may want to tell a judge or jury that IQ scores come with a margin of error. “What’s wrong with that? It doesn’t sound so terrible,” Breyer said.

Florida solicitor general Allen Winsor responded that abandoning the 70 cutoff could double the number of death-row inmates who claim they are mentally disabled. Justice Elena Kagan cut in to acknowledge that Winsor might be right about the number of claims.

“But some of them may be mentally retarded,” Kagan said.

Winsor responded that state law has determined “they are not mentally retarded if they don’t have an IQ score of 70 or below”.

Justice Antonin Scalia pointed to the brutality of Hall’s crime and the several steps it took to abduct and kill the pregnant woman, and a sheriff’s deputy a short time later, as indications that Hall seemed to have sufficient mental capability.

“Could the state show that in refutation of his mental retardation evidence?” Scalia said, using the term for mental disability that both sides use in court papers.

The justice also noted that Hall, 68, has been on death row for 35 years and that he did not raise a claim of mental disability for the first 10 years of his time in prison.

A judge in an earlier phase of the case concluded Hall “had been mentally retarded his entire life”. Psychiatrists and other medical professionals who examined him said he is mentally disabled. School records from the 1950s classified Hall as “mentally retarded” – then the commonly accepted term for mental disability.

Psychiatrists and psychologists who are supporting Hall say an IQ test alone is insufficient for a diagnosis of mental disability. They say there is a consensus among the mental health professions that accurate diagnosis must also include evaluating an individual’s ability to function in society, along with finding that the mental disability began in childhood.

They and Hall also contend that an IQ score is properly read in a range because the results are generally reliable, but not 100% so. The range takes into account a margin of error, a feature of all standardised testing.