The Sun claims that the music impressario wants to recapture the original dynamic that propelled the show to popularity with Sharon Osbourne, Louis Walsh and himself on the UK judging panel.
Dannii Minogue, who was added to the original panel of three in 2007, and Cheryl Cole, who replaced Osbourne in 2008, are also allegedly being considered for a return.
A source close to the show told the newspaper: "The first three series relied solely on genuine British talent and a chemistry between the judges that simply hasn't been there since Sharon left. By series three they had produced an international star with Leona Lewis and the show had real credibility.
"Now it's just a circus freak show with the ringmaster trying to pull the strings on the other side of the Atlantic.
"If Simon can get the original judges and format back he will truly have pulled victory from the jaws of defeat."
A spokesperson for Cole denied that she is in discussions to return, while Osbourne's representative said she "is always open to good suggestions".
Cowell is currently appearing as a judge on the US version of the show, while Louis Walsh remains the only original panel member, now working alongside Gary Barlow, Tulisa Contostavlos and Nicole Scherzinger in the UK.
The show's contact with ITV1 expires in 2013.
Watch X Factor and Strictly Come Dancing stars on the Cosmopolitan Woman of the Year Awards red carpet below:
> 'X Factor': Matt Cardle 'banned from performing on show'
> Lucy Spraggan leaves 'The X Factor'
> 'X Factor' is an insult to people's intelligence, says 'Strictly' Craig