The musician criticised the talent contest for failing to offer its winners career longevity.
He told the Radio Times: "If somebody is dangling this carrot of 'fame and fortune' in front of a check-out girl from Barnsley who can sing like Britney Spears then they're going to go for it, right, because they live in an a***hole - no disrespect to Barnsley."
He continued: "You spend a year slogging around the country and then what? You can't go back to Barnsley and be a check-out girl cos that would look bad.
"So you end up either trying to be an actress or trying to sustain your profile by going on celebrity shows."
Gallagher also questioned why Dannii Minogue was a judge on the show, saying that she "wouldn't know talent if it kicked her in the a***".
"You've got someone like Dannii Minogue telling you you're really talented - and if there's one person in the room who wouldn't know talent if it kicked them in the a*** it's Dannii Minogue," he said.
> Click here to read our interview with X Factor rejects Bad Lashes