Gary Barlow has confirmed that he will be back for the new series, and Tulisa Contostavlos and Louis Walsh are expected to join him, but the ex-Destiny's Child singer has yet to sign up.
It was recently reported that Rowland is holding out for £1.5 million, but an insider has reportedly told The Sun that this is not the main issue.
"There's a chance she could come back, but it looks increasingly slim," the source said.
"Money demands are not the main factor. Kelly didn't connect with the audience as well as people hoped."
The X Factor's open auditions tour around the UK has finished, although hopeful singers can still apply via Facebook until May 8.
A new twist to the show's format means that any artist or act, aged 16 or above, can apply for the show, even if they have a management deal.
Filmed auditions for the programme will begin on May 15.