The Team GB stars are said to be joining double gold medallist Mo Farah at one of the live shows in September. ITV1 has allegedly spent around £1m to secure the athletes' appearance.
"X Factor producers were literally jumping up and down with delight when they heard that all four Olympic athletes had agreed to do the show," an insider is quoted as telling the paper.
"If these wannabes really want to achieve their dreams and win The X Factor then there are no better people to spend time with."
The source also claimed that Simon Cowell approached the athletes just hours after last weekend's closing ceremony, hinting that the special episode could be "one of the most watched X Factors ever".
The reports follow rumours that the show's ratings rival, the BBC's Strictly Come Dancing, has been trying to get Olympians including Rebecca Adlington and Victoria Pendleton on board for its return to screen this autumn.
"Simon can see the irony of using the stars of what was ultimately a BBC event to help boost his own ratings," added the source. "And if his mind wasn't already made up, the moment he heard that Strictly was linked to some of the athletes he knew he had to act fast.
"In Mo, Bradley, Chris and Jess he has bagged arguably the cream of the crop from Team GB, and he is ecstatic they are going to be involved."
The ninth series of the talent contest began last night and recorded the lowest launch show viewing figures since 2006.
Photo gallery - Team GB's gold medal winners:
Copyright: Stephen Pond/PA Wire/Press Association Images