Barlow has been a frequent critic of the 17-year-old since her return to the show on November 12, stating that Lily deserved to be axed by mentor Kelly Rowland in the first week and berating her for "shouting [her] way through" the latest sing-off, in which his own act Craig Colton was eliminated.
Defending his remarks on today's edition of Loose Women, Barlow insisted: "I thought the comment was absolutely right, her performance was very out of control.
"We're watching high emotion on a programme like this and this to me is when stars really shine. When the pressure is on, you really see what someone's about and for me Craig really stole that performance.
"It was very, very understated and actually reeked of experience to me. Hers seemed to be very desperation."
However, Barlow remained adamant that he only gives out "constructive" criticism to contestants.
"I've never felt like I've been nasty," he said. "I really didn't ever want to become someone who says something for the sake of it.
"But I am really serious about music and when I see something on TV I am critical in my head anyway, so I have been used to hearing things and picking them to pieces for years. When I see someone that walks onto a stage, I hope I can be constructive."
The X Factor continues Saturday, November 26 at 8pm on ITV1.