The 'ill Manors' singer ruled out ever appearing on the ITV1 reality show, claiming that he would be "neglecting" his musical gift if he were to use the show as a platform.
"No disrespect to anyone involved in those shows, but all they do is boost your profile as a celebrity," he told Shortlist.
"There are a lot of pop acts with just a pretty face and a half-decent voice. They don't even write their own s**t. I'm lucky enough to have a gift that allows me to create things from nothing: music, films, fictional characters. Others don't have that, so their work goes into their image as a celebrity.
"So, if I have a gift like this, wouldn't I be neglecting it if I appeared on a show like The X Factor?"
Plan B also claimed that the reason he turned down the chance to work with Jay-Z was because it had been set up by industry people rather than personally by the US rapper.
"I want people like [Jay-Z] to come to me and say, 'I like what you're doing, let's collaborate'. Otherwise, I'd just be licking their arse," he said.
"Music should be a personal thing, the minute you start thinking, 'Who can I get in with?' you're basically saying you don't give a f**k about artistic expression. You only give a f**k about money."