The Strictly Come Dancing head judge believes that Cowell should be ashamed of his "spiteful" motive.
Shortly after signing Dixon from Strictly, Cowell said: "With us and the BBC, I laugh about it, but there is unbelievable rivalry now. They want to be number one, I want to be number one.
"The Alesha thing was kind of fun. My decision to poach her was 70-30 on the fact she was on Strictly and the fact I liked her."
Goodman has now told BBC Radio 5 Live: "I would never knock the opposition and I think it is spiteful and mean if that is the case that Simon said, 'I just want to be spiteful towards Strictly'.
"Because why would you want to do that? Why would you want to spoil millions of people's pleasures? It's not the right way to behave, so if that is the case - shame on you, Simon."
> Britain's Got Talent Alesha Dixon: 'I was leaving Strictly anyway'
Watch this year's Britain's Got Talent stars talk to Digital Spy below: