Accordng to the Daily Mail, producers of the ITV reality show see the revamp of the panel as "a potentially catastrophic mistake".
One source said: "Dancing on Ice is a nice family show. But many people tuned in just to hear what outrageous insult Jason would throw at a hapless celebrity after their performance.
© ITV
© Rex Features
"He was box office - and the ratings so far are proving that."
Louie Spence and Katarina Witt replaced Gardiner and Emma Bunton for the 2012 series, leaving head judge Robin Cousins as the only original member of the panel.
Dancing on Ice's ratings slumped to a series low of 7.1m on Sunday, while only 5.9m watched The Skate Off.
Gardiner, 40, recently insisted that he was not sacked by ITV and chose to leave the programme.
> Louie Spence: 'Dancing on Ice is really easy for me'
