Recent reports had suggested that current X Factor judge Cheryl Cole could be involved in the American edition, while Pussycat Doll Nicole Scherzinger and Katy Perry have also been linked to the venture.
However, in an interview with Deadline, the music mogul insisted that the "only person that's been booked" for the Fox competition was him.
"We're trying to keep it as open as possible," he told the website. "People approach me to be on it all the time. It's not about booking well-known people, it's putting together a panel that hopefully Americans will relate to and like."
He added: "There will be some surprises, genuinely some surprises."
Cole has been rumoured as a possible candidate for US show for many months, with Black Eyed Peas frontman will.i.am recently saying that the Girls Aloud star would be "loved" by the public if the speculation was true.
Scherzinger and Perry were reported to have impressed producers with their guest judging stints on the British show earlier this year, while other names who have been linked to the project in the past include Mariah Carey.
The X Factor will premiere on Fox next year.