The 'Treatment' star, who is signed to Simon Cowell's label Syco, said he thinks his boss's talent show is about people who want to be famous.
He told Yahoo!: "I don't watch it - it isn't interesting to me. I think X Factor is… The X Factor, that's it."
However, Labrinth did backtrack slightly when asked about Cowell, admitting he is the best in the business when it comes to finding singers.
"It's not my thing, but if you want to be a singer then Simon Cowell is your man, but if you are an artist you do your own thing," he added.
The 23-year-old became the first artist signed by Syco who wasn't from The X Factor in six years when Cowell signed him in 2010.
Watch the video for Labrinth's single 'Treatment' below:
X Factor latest pictures:
Copyright: ITV Thames TV/Ken McKay