The cyclist also accused the Simon Cowell reality series of dragging down the public mood following the success of Team GB at London 2012.
Speaking about rumours that he may get involved in the music show as a mentor during the Live Shows, Wiggins said: "Let's not talk about The X Factor. Compared to the Olympics, everywhere you went the country was on a high, and as athletes it was phenomenal to see that, then you see X Factor and it's like, 'Oh God, everyone's got to put up with that all winter now'."
Wiggins's music hero Paul Weller has been a vocal opponent of The X Factor in the past, claiming that the show falsely portrays its acts as "edgy".
Fellow Olympian Sir Chris Hoy has also ruled out any involvement in the ITV1 show, tweeting: "Some amusing stories in press about me being a mentor on X Factor. That's the first I've heard about it."
Jessica Ennis has revealed that she would be interested in taking part in the show, but told Daybreak that there had been no official approach so far.
The X Factor continues on Saturday nights at 8pm on ITV1.
[The X Factor - Show One Pictures]
Copyright: ITV Thames TV