The 52-year-old X Factor boss said that he does not feel like he would able to "deal" with the needs of an infant.
"I worry about [kids]," The Sun quotes him as saying. "If they could be born at the age of five, I'd quite like that.
"I'm not sure if I could deal with the 'ga ga goo' thing."
Cowell also reiterated his concern that he will never get married, describing himself as too "weird" to love.
"I'm difficult to live with," he said. "I'm selfish, quite weird - and I need time on my own."
Cowell confirmed his separation from fiancée Mezhgan Hussainy earlier this year, admitting that the engagement was a "big mistake".
Details of past affairs with Dannii Minogue and Alicia Douvall have also come out in recent times.
Photo gallery - Simon Cowell's ex-girlfriends:
Copyright: Rex Features