A recording of Gray and Keys suggesting that female referee's assistant Sian Massey would not know the offside rule was leaked to The Mail On Sunday yesterday.
Keys also made reference to an earlier newspaper column by Brady, wherein she complained about the sexism that exists in football.
Speaking to BBC Radio 5 Live, Brady said: "It makes my blood boil, if I'm absolutely honest.
"What really upsets me is the fact that only females in our industry are judged by their gender. And that is categorically wrong."
The Apprentice star added: "It never would have occurred to me that they had those views, whether public or private, and I'm disappointed.
"It almost makes it worse that they're speaking when the microphones are not on as opposed to when they are on because they'd never really have the brass neck to say it publicly."
However, Keys's sister Susan rang Victoria Derbyshire's show to defend her brother, adding: "He hasn't got anything bad to say about women. I have never heard him making comments before.
"I think it was a bit of banter. I know Richard inside out and he is surrounded by women: he's got three sisters, a daughter and has the greatest respect for women... Perhaps we're all getting a bit precious here."
Keys and Gray have "been stood down" from their duties for tonight's game between Bolton Wanderers and Chelsea and are expected to be replaced for the contest by Dave Jones and Jamie Redknapp.