The Hollyoaks actor, who Gardiner branded a "pillock" earlier this week, said that the show wouldn't suffer if the Australian choreographer was removed from the judging panel.
"I don't think it matters who is on the judging panel. It's the show that's the big thing. It doesn't matter about the contestants or judges," Richardson told DS.
"If he wasn't on the judging panel, I don't think the show would suffer. It's an amazing concept and Chris [Dean] and Jayne [Torvill] are the stars. They made ice skating famous and they made the show. Whether Jason is on the show or not, I don't think it would make much of a difference."
The 24-year-old also claimed that Gardiner's repeated harsh criticism of certain contestants is "unfair".
"When me and Gary [Lucy] were discussing it yesterday, we said that this show is different to X Factor, where they have mentors who are supposed to have their favourites. On this show, the judges should be impartial," he said.
"It's a bit unfair, really. But maybe in the long run he has helped us out."
The Dancing On Ice final airs on Sunday at 7.45pm on ITV1.