As previously revealed, bosses had hoped to round up contestants including former winners Tony Blackburn, Joe Pasquale, Kerry Katona and Phil Tufnell to celebrate the 10th anniversary of the ITV show.
"Trying to convince some of the most outspoken celebrities to spend so much as a night together is proving incredibly difficult," a source told The Mirror. "A lot of the show's former stars absolutely hate each other, and as soon as their names are mentioned together all hell breaks loose. It's a real headache for the show's producers."
Dancing On Ice star Kerry Katona has also expressed her interest in returning to the jungle.
"I'd definitely do it again," she said. "I saw Ant and Dec the other week and thought, 'should I say something?'. I gave my first crown to charity, so I need another one!"
However, she admitted that she would not sign up for the special if Katie Price will be involved.
"No, I wouldn't do it then," she added. "I'll do it if [Peter] Andre does it, and I love Joe Swash, I'd do it with Joe. I love Christopher Biggins too, so I'd do the show with him."