The UK X Factor presenter was in the running for the debut season of the American show but lost out to Steve Jones, who will not be returning for this year's show.
"I was on the shortlist for it, but I signed for the UK one, and as soon as I did I knew it would probably rule me out, even if I thought I could do both in a sort of Superman, bravado way," O'Leary told ShortList.
"It's a shame. I would have liked to have given it a crack because I think I could have done a good job."
Asked if contestants ever give a fake backstory on The X Factor, he added: "With some people, you just get an inkling, like, 'Really? That can't be right'. But I think they are weeded out before we see them.
"When I meet someone and ask them why they're here, I expect them to say, 'Because I'm a great singer', or, 'Because I've never really sung before and will give it a go'.
"But if someone says, 'I'm doing it for her', and raises up a bemused-looking 2-year-old, I sort of think, 'We're all doing it for her, for our families, aren't we?' Maybe you should just do it because you can sing."