The original judges were spotted at the Arts Club in Mayfair, London, last night (February 15), following reports that Osbourne has been approached to make a comeback to the panel.
"The first three series relied solely on genuine British talent and a chemistry between the judges that simply hasn't been there since Sharon left," a source told The Sun.
"By series three, they had produced an international star with Leona Lewis and the show had real credibility.
"Now it's just a circus freak show with the ringmaster trying to pull the strings on the other side of the Atlantic."
Osbourne is thought to have been lined up by producers to replace Tulisa Contostavlos, who is reported to both be facing the axe and planning to quit the ITV show.
Cowell recently held crisis talks for The X Factor, discussing options including a back to basics reboot and Cheryl Cole returning.
He has previously ruled out returning to the UK panel full-time as he focuses on The X Factor USA, which airs on Fox.