The 40-year-old, who mentors Craig Colton and Marcus Collins on the ITV show, claimed that Robbie Williams, Mark Owen, Jason Orange and Howard Donald have been mercilessly critiquing his performance.
"They think I'm rubbish," he told The Sun. "[They] tell me to get back to my day job."
Barlow lost another X Factor act this week when Frankie Cocozza was ejected for allegedly bragging about using cocaine.
At a press conference on Thursday, Barlow accepted that he was partly responsible for Cocozza's behaviour, saying: "We've all encouraged Frankie to not only be good, but to be bad, as well. We've enjoyed it, written about it. The show has benefited from the amount of publicity we've got out of it, but it's just gone too far out of control of the show to keep him here."
Barlow's friend Jason Donovan recently told Digital Spy that Cocozza's exit has "been tough" for the star.
The X Factor continues tonight (November 12) at 8.15pm on ITV1.
> X Factor's Frankie Cocozza: "I've got to stop being a little s***"