The talent show judge was rumoured to be making a return to the British edition of the show in 2013, after two years away while he made the US version for Fox.
However, Cowell apparently denied the reports when asked if he was planning a return at yesterday's Britain's Got Talent auditions in Cardiff, according to The Sun.
A source said: "Simon shook his head. People weren't happy."
[The 2012 'X Factor' judging panel]
The X Factor's judging panel for 2013 has been the subject of much debate in recent months, after the 2012 series saw its lowest ratings since 2006.
[Tulisa at the 'X Factor' 2012 final press conference]
Tulisa Contostavlos recently denied reports that she had quit The X Factor, after discovering that Nicole Scherzinger had already been invited back.
Sharon Osbourne has reportedly already met with producers about a potential return to the show later in the year.
[Sharon Osbourne with Louis Walsh at Judges' Houses]
She was seen helping out Louis Walsh at Judges' Houses during the 2012 series as he chose his final three acts for the Live Shows.
Cowell, Amanda Holden, David Walliams and Alesha Dixon appeared at this year's first Britain's Got Talent auditions in Cardiff yesterday (January 16).
In pictures - Britain's Got Talent auditions