The 31-year-old is thought to have told bosses that she won't be coming back for the second series and picking up her next paycheque of $15 million (£9.4m).
Spears - who sat alongside Cowell, Demi Lovato and LA Reid on the judging panel - failed to impress critics with her "boring" feedback and struggled to "connect" with her contestants - some claiming she should "barely look at them on stage".
An insider said: "Britney and her team knew it was unlikely she'd be asked back as a judge.
[Britney mentoring her act]
"The reality is that towards the end of last season the main players knew they wanted to replace both Britney and LA Reid."
Reid - who quit last month - apparently "wasn't at the top of his game in second season and he knew that".
Other reports claim that Spears has left to concentrate on her own music. She is currently in the studio recording her eighth album and will potentially go back on tour.
A source said: "Most people didn't expect Britney to make it through the season as it's a demanding role but she did her best."
A previous report in Us Weekly claimed that Cowell had paid Spears a fortune to just say "amazing and offer half-claps".
Cowell has already fired judges Paula Abdul, Nicole Scherzinger and presenter Steve Jones last January in a bid to boost ratings.