Soldier Kerr was one of the first four acts voted off the ITV1 show last night, alongside Amelia Lily, 2 Shoes and James Michael.
Barlow was heavily critical of Kerr's rendition of The Kinks' classic 'You Really Got Me' at the weekend, claiming that the 27-year-old stuck out like "a sore thumb".
"I think tonight, you've made Louis's decision really easy," said the Take That star.
> X Factor Results Show: Live Blog
However, Kerr told The Xtra Factor that he believed Barlow's comments were motivated by a fear that he may steal some of the votes away from his Boys category.
"I'll be cocky and say that because I'm not the ugliest person in the Overs and Gary didn't want all the female fans fancying me," said Kerr.
"He didn't want me splitting the female fans with Frankie."
The Birmingham singer also claimed that he was disappointed not to be given a second chance on the reality show, arguing that he may have lasted longer with a public vote.
"I'd rather the public sent me home," he said. "I had one bad performance on the live shows. I'd like to have had two at least. I think I could have brought something. I'm nothing but nice to people. I hope the image I've put out is that I'm a nice person."
The X Factor continues next Saturday at 7.45pm on ITV1.