The ex-Oasis songwriter told Key103 that the show boss wanted him to be his direct replacement on the reality talent contest.
Gallagher said: "He was saying we're going to revamp the show... and he was saying that we need somebody to - how he put it - be the alpha male.
"He said, 'Basically somebody to replace me', and I was like 'What, you mean have all the kids round my house in the back garden and saying to big lunatics 'I've made my decision and I can only take three...'.
"No, that's not going to happen in my house, the missus would kick off for a start and I've got a 4-year-old lad who's on a power trip at the minute and he'd be like, 'You're rubbish, shut up!'"
He added: "I like Simon Cowell, I've met him countless times out and about and he gets a bad rep for the music and all that, but if the music business was inhabited by more people like him it'd be alright, because he's honest.
"He's not claiming that he listens to The White Album every night or Led Zep IV before he goes out. He likes what he likes and he's passionate about that and good luck to him.
"He's a nice guy, but I was like, 'You don't want me on there', and he was saying we do, and I was saying you really don't want me on there, because I will just put through obese people... or midgets."
Gallagher is being sued by his brother and former bandmate Liam for his alleged "lies" regarding the break-up of the band.
Watch the Oasis 'Where Did It All Go Wrong?' music video below: