Host Steve Jones and judges Paula Abdul and Nicole Scherzinger were axed from the Fox musical competition in what has been described as a "natural reassessment" of the show's approach.
Following the shake-up, rumours have circulated regarding Abdul and Scherzinger's replacements, with Mariah Carey and Beyoncé suggested as possibilities.
However, in an interview with Extra, Cowell claimed that spending money on big stars is not worth it in the long run.
"When we first launched [Pop] Idol in the UK years ago, we didn't think for one second there had to be an A-list star on the panel. Everyone on the panel was really unknown. Firstly, they are incredibly expensive - that is millions you lose on something else," he explained.
"Secondly, they have to make the time commitment and on my show, they have to work five days a week. Thirdly, they have to be interesting… I'm not playing this game where we have to compete on the level of A-lister stars… you can lose focus on what the show is all about, and at the end of the day the most important thing is the contestants."
As for Carey and Beyoncé's alleged involvement, Cowell said: "I read that online that I offered [Beyoncé] $100 million (£63 million) a year. The budget for the show is $100 million. That would never, ever happen. It's absolute nonsense… Even if you had that kind of money. I wouldn't spend $100 million on one person.
"I would love [Mariah]. But you can't expect someone who's just had two kids to make the kind of commitment you would need on this show."
Cowell will return to The X Factor USA later this year along with panelist LA Reid, who recently signed a contract for season two of the Fox show.