Britain's Got Talent producers apparently told the Little Britain star to also refrain from teasing his boss about his height and campness while on the panel.
Walliams told The Sun: "I meet the producers before Simon arrives and I say, 'Can you make jokes about Simon?' and they go, 'What, what kind of jokes?'
"I said, 'Well, I dunno. Jokes about his height?' 'Absolutely no way!' 'What about jokes about his friendship with Sinitta?' 'No!' 'Jokes about him being camp?' 'He detests jokes about him being camp'."
Speaking of a Cardiff audition, he added: "The first person comes on the stage and they say, 'Oh, my dream is to perform in front of the Queen'. I say, 'Well, he's right here, go ahead!'"
Walliams recently admitted that he initially found it difficult to buzz Britain's Got Talent contestants.
Watch the Britain's Got Talent judges talk to Digital Spy below: