Clark, who regularly clashes with the judge on the show, told The Sun that while he is happy to exchange "panto" insults, he believes Barlow's comments have become "hateful" after he slammed the singer at a recent gig.
> 'X Factor' Rylan hits back at Gary Barlow with Chris Maloney slam?
The Take That star told fans at his Mencap charity concert on Tuesday night that he thinks Clark is "absolute s**te" and encouraged them to chant "Get him out".
"We've got this banter in the studio and in the studio that's where it stays," Clark said. "So when I found out what he did at the gig it upset me because I thought, 'You've taken this out of the show now'.
"I'm not saying the word 'bully', not in the slightest bit. I just look at it and think it's not nice to get a crowd to do something like that... ever.
"It's a bit hateful. I'm all up for the panto but these people in the crowd are hardcore Gary fans. If he turns round and says something, that's in their head and they're the ones who come on Twitter to give me abuse.
"I get a lot of 'I Love Gary Barlow' people tweeting me and that's expected, but I don't think that helped this week.
"Without going into too much detail, if he does it again and carries on I'll just have to tell everyone how it really is. I'll lay it on the line."
Photo Gallery - Famous X Factor feuds: