During Sunday's Britain's Got Talent auditions at the London Palladium, Holden was left unimpressed by her fellow judge's dig in front of the audience, reports The Sun.
Holden stated that she had gone from "rags to riches", leading Cowell to retort: "You went from Les Dennis..."
She replied: "That's a bit close to the bone, Simon," before adding: "You've pissed me off. We're meant to be friends."
Cowell soon apologised for his comment, telling her: "I'm sorry, I didn't mean it."
> Britain's Got Talent judges witness fire diving audition - pictures
Amanda Holden divorced Les Dennis in 2003 after eight years of marriage, following her affair with actor Neil Morrissey.
A source said: "Simon made the comment about Les Dennis but regretted it straight away so he apologised. Amanda was not impressed."
Simon Cowell is said to have been even more ruthless than usual with Britain's Got Talent acts this year, including dismissing the first 10 acts at the auditions in Cardiff.
In pictures - Britain's Got Talent auditions in Cardiff: