The X Factor boss made the light-hearted jibe as the judges debuted BGT's seventh series in London, an event that Holden arrived at carrying a pair of Union Jack boxers.
When Holden again brought up Cowell's penchant for going commando at a screening of BGT's first episode attended by Digital Spy, Cowell replied: "I'm not discussing that. I think you're the one who's known for not wearing knickers, Amanda."
The comment drew an audible gasp from those in attendance, but Holden - who famously cheated on ex-husband Les Dennis with Neil Morrissey - rose above it, saying: "I can prove I'm wearing pants, can you?" and pulling up her "sexy little pants" above the top of her trousers.
Later, when Cowell was asked if he's ever demanded that footage of himself be removed from Britain's Got Talent, the 53-year-old revealed: "To be honest with you it's Amanda's stuff we have to cut out.
"[For More Talent] we asked her to demonstrate how she would eat a banana, and I tell you, it's one of the best pieces of TV you'll never see. It's outstanding."
Unfazed, Holden pointed to the rings on her fingers and declared: "It's why I've got lots of diamonds. I earned them."
Britain's Got Talent returns tomorrow (April 13) at 7pm on ITV.
Photo gallery - Britain's Got Talent's first contestants: