Simon Cowell was recently quoted in Tom Bower's book Sweet Revenge as describing Barlow's appearance on The X Factor as "too scripted". It was also claimed that he was upset with the Take That singer's OBE award.
Barlow joked that he would respond in kind with his own tell-all book, but said that Cowell has never gave him an impression of any ill feeling towards him.
"Wait until my book comes out! It will be bloody beautiful. I'll get the lot of them," Barlow told The Sun.
"I've had plenty of abuse in the past 22 years. I'm so used to having people saying things about me - then when I actually meet them, it turns out they've never said the things. I take it all with a pinch of salt.
"Gossip and tittle tattle, I've got no time for that."
Watch Gary Barlow talking to Digital Spy about the first X Factor live shows below:
On Cowell's presence on The X Factor, he said: "We get messages every couple of weeks. He gets the show sent to him. He approves it.
"I don't get advice. We get feedback on the shows just saying, 'We love the pace of the show' or whatever.
"He's the executive producer on the show so he makes changes to the edits but yes, we've had good feedback."
Photo gallery - 'The X Factor' final 12:
Copyright: THAMES TV/SYCOTalking about this year's reported drop in viewing figures, Barlow added: "I don't know about audience figures. To be honest, it's not my problem. They are not going to get more audience figures because my critique is better. It's just not my problem.
"My complaint at the end of last year is I thought the contestants weren't good enough. I thought we had a pretty average year last year and coming back this time I am really stoked. I think the show has been great this year. Ultimately people are tuning in for the talent.
"I don't know what's going on with the figures, but if they are going back up it's because people are seeing talent on there."
Barlow recently compared the X Factor stage to the Death Star in Star Wars.