Lily landed in the bottom two on last weekend's show and sang Lady GaGa's 'You and I' for survival, remaining in the competition ahead of Barlow's act Craig Colton.
Barlow told her she shouted her way through the performance and later called her singing "desperate" and "out of control" during an appearance on Loose Women.
Lily is said to have demanded a meeting with the judge in his dressing room following the comments, resulting in a "heated meeting" during which she asked: "Why do you keep putting me down?"
"Gary's comments were hurtful as she respects his opinion, and they played on her mind. Her confidence was very low," a source told the Sunday Mirror.
"Gary told her his loyalty is to his acts and sometimes he has to be critical of other contestants. Amelia seemed to understand and left the room happier."
Barlow gave Lily more positive feedback for her performances of T'Pau's 'China In Your Hand' and Kelly Clarkson's 'Since U Been Gone' last night, telling her: "You came back with a great performance."
> T'Pau's Carol Decker calls Gary Barlow a t**t over 'X Factor' comments
Watch Amelia Lily performing Lady GaGa's 'You and I' on The X Factor below: