The X Factor judges' alleged affair has been revealed in extracts from former BBC journalist Tom Bower's book The Intimate Life Of Simon Cowell published in The Sun.
"I had a crush on her. It was Dannii's hair, the sexy clothes and the t**s. I was like a schoolboy," he is quoted as saying.
"She was foxy. She was a real man's girl. Very feminine. It was genuine love."
The short-lived relationship is alleged to have taken place in 2007, after Cowell split from Terri Seymour but before Minogue became involved with Kris Smith.
The singer apparently caught Cowell's eye when he added her to the X Factor panel, describing her as his "new toy".
"Every guy in the world fancies Dannii. She's become a sex symbol since joining the show, and the contestants love her. She's got that Sophia Loren, movie-star sex appeal, I love sitting next to her," Cowell said of his then new recruit.
The pair denied the romance when rumours first surfaced in 2008. Cowell became "mesmerised" with Cheryl Cole once his fling with Minogue cooled, the report adds.
Cowell, who reportedly enjoyed Minogue's feud with Sharon Osbourne, is also said to have tried it on with Natalie Imbruglia as recently as last year, but only got a kiss and a cuddle.
Minogue was recently reported to be "inconsolable" after splitting from Smith.