Forget “Who shot Emily Thorne?” The real question hanging over Revenge‘s third season was, “Um, what’s the deal with Victoria and her hunky son?”

We’d learned in Season 2 that Victoria Grayson, regally played by Madeleine Stowe, had given a child up for adoption years before. And when reappeared in Queen V’s life as an adult, he looked a lot like Justin Hartley (This Is Us).

Their relationship, especially in the beginning, repeatedly sounded an odd note: Were Patrick and Victoria flirting with each other?! (Or, as TVLine editor in chief Michael Ausiello put it in Spoiler Alert! at the time, “They’re sexy together, the two of them.”) The fact that Hartley spent a good amount of time in his first episode wearing nothing but a towel added to the so-wrong-it’s-right feel.

When we recently raised the issue with Hartley, the This Is Us star knew exactly what we were talking about — and said he missed the Oedipal feel when it disappeared from the scripts later that season.

“I don’t know if it was dropped on purpose or not,” Hartley said, laughing. “When I read those and talked to Madeleine about this in depth — she’s one of my favorite human beings, I love Madeleine — we were reading it and I thought, ‘Am I reading something into this?’ Then she said it separately… she goes, ‘That’s definitely a weird sort of thing.'”

As it turns out, Hartley has given the matter a deal of thought over the years.

“I mean, here’s the mother who sent her son away and then found him again. Her children at the time didn’t like her, so she’s looking for something. She hadn’t been with him his entire childhood so she only knows him as a man,” he said. “So that’s kind of a weird dynamic. I thought it was in there a little bit. Maybe [the writers] didn’t realize it was in it, and they were like ‘Oh, s–t. We better stop writing this.'”

For the record: “I thought it was fantastic,” Hartley says, chuckling. “I was like, ‘This is creepy and excellent.’ I loved it.”

So there you have it! Watch some scenes from Patrick’s first episode below, then weigh in on Hartley’s Revenge reminiscing, then hit the comments with your thoughts!