The ex-Oasis songwriter recently confirmed that he had been approached by Simon Cowell to fill the "alpha male" head judge role on the show.
Gallagher admitted that his daughter has not forgiven him for turning down the position after he told Cowell that "he didn't really want me on there".
Asked about the approach, Cowell's eventual replacement Barlow told The Big Issue: "I don't know about [Noel being a good judge]... It's a big commitment.
"If you're going to get involved with something like this you've got to be prepared to give a lot of time and effort to it… I can't see him doing it somehow."
Of suggestions by some, including Gallagher, that the 16-year-olds on the show are too young, Barlow countered: "It's a massive opportunity. I think you've got to want it this bad. In fact, my criticism is, some of them should want it more than they do.
> Noel Gallagher 'turns down X Factor offer'
> Noel Gallagher: 'Music industry better with more Simon Cowells'
"The pop world is a bit like fashion. It moves very quickly. What happens with some people is, they earn a bit of money and they think, 'I'm all right now, and I can't be arsed with all the hard work anymore'."
Regarding the media attention for The X Factor, Barlow said: "The press stuff is
"It's kind of part and parcel of the whole show so you learn to live with it. I don't read newspapers but you hear the whispers and you think, 'Jesus Christ, what is happening... what is happening here?' The truth is it's probably keeping a lot of the papers going."
The Big Issue is sold by homeless and vulnerably housed vendors for £2. £1 of the cover price goes directly to the vendor. The full Gary Barlow interview features in this week's issue.