The singer - who has previously served as a judge on the show - declined an offer believed to be in the region of £1 million yesterday (Thursday, May 17) and it has been alleged that she was seeking a fee of up to £5 million.
> Dermot O'Leary confirmed to host 'The X Factor'
"We really wanted Dannii, but it has been a slow and drawn-out process and we don't have any time to lose," a source told The Mirror.
"We need to find someone as soon as possible, and so we decided Dannii had enough time and was offered enough money so we walked away from the deal."
The first auditions in front of the judges are due to take place on Wednesday, May 23 in Liverpool, with The Sun reporting that guest panellists will fill in until a permanent judge is hired.
> Katy Perry, Lily Allen, Noel Gallagher: 'X Factor' replacement in pictures
Louis Walsh and Gary Barlow will both return to The X Factor panel, and former N-Dubz member Tulisa Contostavlos is also expected to return.
Minogue left The X Factor in 2010 and had recently been linked with a return to the ITV1 reality show following Kelly Rowland's departure.
She recently spoke of a reluctance to disrupt her young son Ethan's "little routine" in a magazine interview.