The author of the controversial book Sweet Revenge told Heat that Cowell would have liked Cole and Minogue to join Louis Walsh and Gary Barlow on the 2012 panel.
"Simon will have been very pleased," Bower said of Cole's guest spot with Barlow for the Judges' Houses round.
"He always wanted her back. That was the whole problem. will.i.am prevented her, in his view."
He added: "Simon thinks The X Factor is the natural place for her, and his ideal lineup this year probably would have been Cheryl and Dannii with Gary and Louis.
"He would have very much liked Dannii there - he thinks that would have brought in a lot more viewers and it would have been a better show."
A source previously quoted Cowell as telling ITV director of television Peter Fincham: "Bring her back to X Factor... everyone will be looking at her tits."
Bower claimed in the updated version of Sweet Revenge that Cowell wanted Minogue to replace Kelly Rowland, but that she wanted £5 million while he offered £2 million.
Asked why Nicole Scherzinger was picked for this year's show, he added: "There was no-one else."
Of Cowell's own potential return to the UK show, Bower said: "There's no doubt he thinks he'll come back for the final. The role that he'll play will become clearer in a few weeks."