The Mirror claims that bosses are growing tired of Simon Cowell's indecision over who will sit on the judging panel for the new series.
Cowell previously confirmed that Take That's Gary Barlow will join Louis Walsh - who is expected to return to his position. However, it has been suggested that there are fears that possible replacements Tulisa Contostavlos and Kelly Rowland will disappoint viewers.
"She is not a big enough name," one boss allegedly said of the N-Dubz star. "She is not another Cheryl Cole or a Dannii."
An insider continued: "ITV wants things signed and sealed but Simon is still changing his mind and thinking of new names.
"The feeling is that without Cheryl and Dannii a big sexy female star is needed. If they end up with Tulisa and Kelly Rowland, who are the favourites for the jobs, fans will be very disappointed."
Minogue announced last weekend that she had quit the show due to clashes with Australia's Got Talent. Her decision saw scheduled auditions in Manchester postponed until next month.
Responding to the claims, an ITV spokesperson said: "We are very excited about this year's show. We'll announce the judges' line-up shortly."
> Dannii Minogue: 'I'll still watch The X Factor'