The Syco boss is said to have turned up to the Birmingham auditions more than an hour late, leaving the crew and audience fuming.
A show source told the Daily Star: "Everyone was raging. The show was scheduled for 3pm.
"The presenters were ready, thousands of fans were waiting and Simon hadn't even left London yet."
David Walliams, Amanda Holden and Alesha Dixon have returned to the panel alongside Cowell. They are currently filming auditions across the country, with the final round taking place this weekend.
Cowell recently missed five hours of the Manchester auditions while suffering from a migraine.
Holden joked: "It's ace when he isn't here. We can have our say and relax and drink lots of champagne at his expense."
Watch Simon Cowell, Ant & Dec and the BGT judges chat to Digital Spy below: