If Cowell does have a son, he's already got a track record playing a fatherlike role to the five boys we now know as One Direction.

On Tuesday's Chelsea Lately, he talked about how Harry Styles, Liam Payne, Niall Horan, Louis Tomlinson and Zayn Malik were all eliminated on their solo X Factor auditions, but he saw their potential anyway.

"I liked these boys, and I thought let's give them another shot," he said. "And we put them into a band."

And he's already had a taste of kids (OK, young adults) misbehaving. "With these five boys, you can't tell them what to do," he said. "They make all their own decisions."

Maybe that's true, but if they ever act out of line, you can bet Simon will show some tough love!

—Additional reporting by Melanie Bromley