The show creator tipped the One Direction pair to join the panel on a future season, following their stint as guest mentors on the Australian version.
After footage of their X Factor Australia appearance was screened at the X Factor USA launch yesterday, Cowell was asked which One Direction member would make the best permanent judge.
"It would probably be between Louis and Harry," he replied, before quipping to fellow judge Demi Lovato: "Demi, you would probably choose Niall [Horan]."
Cowell previously teased Lovato about her rumoured romance with Horan in backstage footage of The X Factor USA.
"Keep away from him. He's pure," he joked, before asking if the pair have kissed. Watch the clip below: