The 'Earthquake' singer is signed to Simon Cowell's Syco record label, and has reportedly been instructed to sing on an upcoming live show.
He explained that while he does not hate the show, he doesn't agree with its values and would prefer not to be involved.
"I've been forced," he is quoted as saying in The Sun.
"A lot of people think I hate X Factor and think it's s**t. I'm just being honest about the way it works. I think artists who take their craft really seriously don't go on it."
Labrinth recently criticised The X Factor, stating that the show "isn't about musicians".
The 23-year-old became the first artist to join Syco who wasn't from The X Factor in six years when Cowell signed him in 2010.
Watch the video for Labrinth's single 'Treatment' below: