The Olympians join previously-announced celebrities Richard Arnold, Kimberley Walsh, Dani Harmer and Nicky Byrne on the show, BBC News reports.
Pendleton retired from professional cycling at the age of 31 after winning a gold and silver at London 2012.
Smith, 23, won a silver medal in the pommel horse after winning a bronze in Beijing four years ago.
He auditioned for The X Factor in 2009 as "a bit of fun" after previously picking the gym over a singing scholarship when he was 8 years old.
"I made it through to the fourth round of X Factor, which is when I get to appear on television and meet Simon Cowell and the rest of the judges for the first time," Smith told The Daily Telegraph at the time.
"I just really want to hear what Simon says about my voice whether he thinks I could make it as a solo singer."
Strictly Come Dancing 2012 - rumoured celebrities
Copyright: PA Images Dominic Lipinski/PA Wire