Before Ben Affleck appeared on The Late Show this week, he probably knew what he was walking into. Given the slew of sexual misconduct accusations and revelations that have been swirling through the entertainment industry—and Affleck’s position as an early career acolyte of Harvey Weinstein’s—the topic was bound to come up. Still, as Colbert shifted toward the disgraced producer during their interview Thursday, Affleck dryly joked, “This is a comedy show, correct?” What followed was an apparently candid conversation that, in reality, left out a few crucial details.

It’s understandable that Affleck—or anyone else—would be uncomfortable discussing the rash of allegations that have torn through Hollywood. But for Affleck, the topic hits particularly close to home: as Colbert mentioned, the actor was closely affiliated early on in his career with Weinstein, and has himself been accused of groping multiple women.

Affleck noted that he’s donating his Miramax residuals to charity. With regard to the allegations against him, Affleck said, “What I was accused of by a woman was of touching her breasts while I gave her a hug. I don’t remember it, but I absolutely apologize for it. I certainly don’t think she’s lying or making it up. This is just the kind of thing that we have to—as men, I think, as we become more aware of this, be really, really mindful of our behavior and hold ourselves accountable and say, ‘If I was ever part of the problem, I want to change. I want to be part of the solution.’“ The woman was Hilarie Burton, to whom Affleck issued a public apology. Colbert did not follow up to ask about the other women that have accused Affleck of groping them, which he has not publicly addressed—and neither he nor Affleck mentioned that his brother, Casey Affleck, was sued by two women for sexual harassment allegedly committed during the filming of I’m Still Here. (Both suits were settled in 2010.).

“It’s just a tricky thing to try to handle,” Affleck said, referring to sexual misconduct. “I think the most important thing to do is to support the voices that are coming forward, believe them, and create a business where more women are empowered and in place so that less of this happens, and so there is a way of reporting this stuff that people can feel safe doing it . . . I’m not a spokesman; I’m not a superhero. I can’t change it by myself. I can just be accountable for myself and for my actions.”

Uncomfortable conversations seem to have become the theme of Affleck’s Justice League’s press tour. In addition to his Late Show appearance, Affleck also spent a good chunk of his Today show appearance Friday discussing the same subject; earlier this week, he also briefly inspired controversy with an awkward joke about Hollywood’s harassment problem that landed with a thud. Then again, given some of the reviews for Justice League, the film itself might not be the safest subject either.