The Black Eyed Peas singer was said to be in early stages of talks to sign up for the Fox talent show to replace one of the show's departed judges - Nicole Scherzinger or Paula Abdul.
> LeAnn Rimes 'X Factor' USA talks confirmed by Eddie Cibrian
However, she has now explained that she is unable to commit to the series after her managers discussed a potential deal.
"I have no idea how this keeps coming back, it's crazy!" she told MTV News. "I know that my managers and agents have been in touch with the X Factor both last season and this season, but both times it's not the right time."
Fergie was also rumoured to be taking part in the first season before she publicly denied her involvement.
Britney Spears has also been linked to The X Factor's judging panel, but negotiations have reportedly stalled after the 'Toxic' singer demanded a $20m contract and a one-year exit clause.
Simon Cowell recently ruled out Glee's Darren Criss joining the judging panel, saying that he did not know him.
The second season of The X Factor USA airs on Fox in September.
> Simon Cowell: 'I'd work with Paula Abdul, Nicole Scherzinger again'
Watch Fergie discuss The X Factor USA below (US users only):