Dannii Minogue has recently been linked with a return to the talent show since her exit in 2010, and is said to be in early negotiations with show bosses.
The Australian singer would replace the outgoing Kelly Rowland, who confirmed her exit from the series yesterday (April 30).
A source close to production told The Sun: "Kelly was never given a contract to look at by bosses. Simon has remained a big fan of Dannii's and wants her back."
The reports come two weeks after Tom Bower's biography of Cowell revealed that he and Minogue had a secret affair during The X Factor's 2007 run.
Kelly Rowland cited "global commitments" and the show's "increasing demands" for her exit.
A show insider added: "Kelly put a lot of people's noses out of joint last year due to her lack of commitment and her demands."
Gary Barlow has reportedly signed a £2 million deal to return as a judge, while Tulisa Contostavlos and Louis Walsh are expected to sign contracts soon.