The DJ, who has spent 35 years in the industry, was recently tipped as the producers' first choice to replace Simon Cowell on the show's panel. Cowell announced last month his intention to quit the hit reality show in order to concentrate on his American version of The X Factor, which is to air this year on Fox.
However, senior television editor Marc Berman claims that the move is "not without its risks", and could "cheapen the brand", also questioning how much Stern knows about music.
He explained to Us Weekly: "It’s an enormous gamble. He certainly is a name. He certainly is not afraid to speak his mind. He would create a lot of controversy. If they’re looking to keep the show on the map, it’s not a bad idea.
"I’m sure he wouldn’t completely get along with Kara or Randy or Ellen. It would be chaos and turmoil."
With speculation already rife over Cowell's successor, an Idol source told Entertainment Weekly that there would be no rush into making a decision.
"The search has just begun, nor are there any frontrunners at this time. Nobody’s going to be offered a job [as a permanent Idol judge] for a while. We’re doing a very thorough search."