The two Britain's Got Talent judges were being interviewed on Ross's ITV1 chatshow to promote the latest series of the contest.
Cowell's annoyance is said to have started after Walliams joked: "The main question I get is, 'Is Simon gay?' Even my mum asks. She says, 'He's nice. Is he gay?'"
Cowell responded that he was "fine" being single, but that many women had said no to marriage.
Walliams quipped: "But asking the whole of Stringfellows all at once, they don't find it that flattering."
Simon Cowell later tried to change the subject by saying that he likes to get up at midday and work until 5am, but Walliams said: "At 12 o'clock at night you're working quite hard - because that's when the ladyboys come round."
Cowell replied to his fellow judge: "It's worth pointing out that I have a relationship with ITV and most of this will be edited out."
The 52-year-old was heard to have told Ross during an advert break: "I am gonna have a word with you," reports The Mirror.
David Walliams later held up a photo of Cowell, saying: "This proves that Simon does like me. Look at how erect his nipples are."
Cowell later stood up in an attempt to stop the interview after Ross made a joke about him touching Walliams's "buzzer".
An audience member said of the interview: "He is not used to being the butt of everyone's jokes."
However, another insider close to Cowell said: "He knows the sense of humour David and Wossy have."
Cowell recently said that he wants a 90-second delay during the live finals due to the uncertainty of David Walliams.
Britain's Got Talent returns for its sixth series this Saturday (24 March) at 8pm on ITV1.
> Britain's Got Talent David Walliams banned from making Sinitta jokes
Watch David Walliams talking about Simon Cowell's sexuality at the NTAs: