Fans of The Vampire Diaries‘ Damon/Elena and Stefan/Elena, brace yourselves: Both the current couple and the erstwhile pair are looking at bumpy roads ahead.

After finding out that versions of Stefan and Elena have been drawn to each other over the centuries, Damon made it clear that he wasn’t going to allow the universe to get in the way of his romance with the Whitmore college freshman. But that’s easier said than done.

RELATED | The Vampire Diaries Season 5 Spoilers — Elena, Katherine and Amara Scene

“It’s the elephant in the room,” describes Nina Dobrev during a TVLine visit to the show’s Atlanta set. “When you’re in a relationship and there’s an ex-boyfriend around, the significant other is always like, ‘Hmm, I wonder…’ There’s always that insecurity. They might say that it’s OK, but it’s in the back of their mind. It’s the same thing with Elena and Damon. He’s pretending to be OK with it, but he can’t unthink and unknow what he heard and what he knows now.”

Having that little destiny nugget tucked away in Damon’s mind “will always cause tension in their relationship moving forward,” continues the actress.

“Relationships are hard in life. Relationships are really hard in Mystic Falls,” adds Ian Somerhalder. “If you don’t have conflict, you have no drama; and if you have no drama, you have no TV show. So relationships don’t last that long in Mystic Falls, unless, clearly, you’re a freakin’ doppelganger and the universe draws you two together.”

RELATED | The CW Picks Up The Originals, The Tomorrow People and Reign for Full Seasons

Damon might be secretly concerned about his girlfriend’s pull to her ex/his brother, but the question remains: Does Stefan even want to be with Elena right now? “I think their love has faded,” admits Paul Wesley. “It’s dissipated.”

The universe, though, might have other plans. “It’s kind of like when you go to a fortune teller and they tell you, ‘You’re going to end up with this person.’ And you’re like, ‘Really?’ And you can’t help but in the back of your mind think that the fortune teller might be correct,” Wesley says of the intel about Stelena’s decades-spanning romance. “It’s one of those things, even though you don’t really see yourself with that person.”

As the younger Salvatore’s mindset changes – “Right now, Stefan’s taking some time to himself. He’s 160 years old, so maybe he’s kind of over relationships?” muses Wesley – his connection with big bro Damon has taken a turn, too, to a place where “five seasons in, it’s not really contentious at this point anymore.”

Oddly enough, the historically at-odds brothers aren’t even battling over Elena’s affections anymore, adds Wesley. “[Maybe] a little bit, but not really… That triangle is, to be honest, a bit on the backburner right now.”

Will all that change now that Stefan has his memories back? It actually seems unlikely that the vamp will be drawn back to his old love just because he remembers her. “There would have been more of a shot of them getting in the sack if he didn’t have his memories,” says Wesley with a laugh. “That’s usually how it works. ‘Let’s forget about everything that’s happened. Let’s get busy.’”

Meanwhile, there’s another factor complicating this already messy romantic triangle: Katherine, who has “lingering love for one of the brothers that we learn more about as episodes progress,” reveals Dobrev. “She does love one of the brothers. Almost like Elena has chosen, Katherine has chosen and we see that unfold.”

The Vampire Diaries airs Thursdays at 8/7c on The CW.