Rex Features
The Welsh singer, who took part in the show in 2004, said Sergeant should not have pulled out of the series because the public would have voted him out eventually.
"I think it's entertainment at the end of the day and some of the comments about him were pretty brutal," Jones told DS. "But every year there has been some slightly dud dancer who has stayed in longer than they should.
"That's just part of the British psyche. It’s all about the underdog. But when it came down to it in the quarter-final or semi-final I'm sure the public would have said they didn't want him anymore.
"That would have been that and everyone would have loved it. Too much fuss was made out of it really."
Jones is currently promoting festive Children In Need single 'Little Drummer Boy/Peace On Earth', which features Radio 2 DJ Terry Wogan.
Strictly Come Dancing's semi-final takes place on Saturday on BBC One.
