The 24-year-old, whose fellow band member Tulisa Contostavlos appears as a judge on Simon Cowell's competition, told The Mirror that working to make N-Dubz a success had given the group "credibility".
"My problem with X Factor is that you come in as a nobody and then the next day you sell over a million records and you haven't necessarily done any of the work to go with it," Fazer complained. "It's weird. We worked 12 years to get to where we are.
"Heartbreaks, broken promises, ups and down - tears and smiles. You name it, it has happened. It has made us the people we are and any credibility that comes with it."
Despite his criticism, Fazer admitted that N-Dubz could have used the format if they had not become successful.
"We might have tried that way instead. It gives people the chance to get on stage in front of a live audience and important industry people and get people to hear their sound," he said. "Also, there wouldn't be people like JLS and there wouldn't be no Leona either."
> N-Dubz Dappy: 'Tulisa Contostavlos will be brutal on X Factor'
> Tulisa Contostavlos: 'I always wanted to be an X Factor judge'
Watch an exclusive clip of N-Dubz performing a mash-up of Jay Sean's 'Down' and Black Eyed Peas' 'Meet Me Halfway' below: