The exercise guru has been a popular guest on Letterman's late-night shows for many years, with their occasional on-air clashes becoming viewer favourites.
Simmons has now revealed to Men's Health that he has never been able to become friends with Letterman in spite of their long history.
"I love David. He's more complicated than any of them, that David... You know, I've never actually met him," Simmons quipped.
"You do the show with him, and when they go to commercial, they remove you. You're just not allowed to talk to him. Or if you are, he's very standoffish."
Simmons last appeared on The Late Show with David Letterman in 2008, but cautioned that he's unlikely to be guest starring on the series again any time soon.
"Maybe one day, when the time and everything is right. Because I do love him," he said. "The last time I was on, we went to commercial and I was like, 'I'd love to meet your wife!' He was like, 'You never will'. And then I was like, 'And see your son!' And he was like, 'You'll never see him'.
"And then we come back from the commercial and he's all friendly with me again... You think I'm kidding, but I'm not making this up."
The Late Show with David Letterman airs weeknights at 11.35/10.35c on CBS.
> Richard Simmons denies illness rumours
Watch a clip from one of David Letterman's interview with Richard Simmons below: