The soap star, who Gardiner described as a "pillock" this week, said that he thought Gardiner was over-stepping the mark when he became personal in his comments on the show.
"I try not to think too deeply into it. I do find when it gets a little bit personal, when he does that I find it hard to explain it to my family," he told DS.
"At the end of the day it is a family show and when I've got my little kid there, who is crying saying, 'Why is that man being so mean to my daddy?' then it goes a little bit too far. But, what can you do? That's Jason and there's only one of him. There's millions watching at home.
"You can't drive yourself mad trying to please Jason. You're on a hiding to nothing. At the end of the day, as long as the people at home are enjoying it, that's all that matters."
Speaking about Gardiner's remarks earlier in the series, when he compared Sharron Davies to "faecal matter", Lucy added: "Again, family. She's got her kids there. Is it necessary? Is there anything she can do with those comments? I don't think so."
However, Lucy insisted that he wouldn't want to see Gardiner removed from the programme, claiming that there would be a "void" without him on the panel.
The Dancing On Ice final airs on Sunday at 7.45pm on ITV1.