Supreme Court Justice Ruth Bader Ginsburg thinks that the Supreme Court’s conservative attitude on abortion access can be traced to one of her colleagues.

“To be frank, it’s one person who made the difference: Justice [Anthony] Kennedy,” Ginsburg told Elle. “He was a member of the triumvirate used to [reaffirm] Roe v. Wade in the Casey case, but since then, his decisions have been on upholding restrictions on access to abortion.”

In Planned Parenthood v. Casey, Kennedy was one of three justices who authored the Supreme Court's majority opinion upholding a woman’s right to an abortion. But the majority also ruled that states could impose certain kinds of restrictions on the procedure as long as they did not place an “undue burden” on the mother. In 2007, Kennedy wrote the majority opinion in Gonzales v. Carhart, in which the Supreme Court ruled 5-4 that a federal ban on partial-birth abortions was constitutional.

Ginsburg, who helped found the ACLU’s Women’s Rights Project and earned praise for her blistering dissent in the Supreme Court’s Hobby Lobby case in June, told Elle that 50 years from now, people would no longer understand the case. She said that there isn’t much activity around pro-choice issues now because many women who need an abortion have the means to get one.

“The impact of all these restrictions is on poor women, because women who have means, if their state doesn’t provide access, another state does,” she said. "I think that the country will wake up and see that it can never go back to [abortions just] for women who can afford to travel to a neighboring state."

Ginsburg, who is 81, also pushed back against calls for her to step down before President Barack Obama leaves office, which would allow him to appoint another liberal justice to the Supreme Court. Ginsburg said Senate Republicans would use a filibuster to block anyone Obama would pick.