Cotton took over the talent show spin-off series from Ben Shephard last year, but has decided not to renew her contract because she's keen to establish herself in America.
According to The Sun, Cotton may have not been offered a new contract by the ITV reality show anyway.
A source said: "Fearne had fun working on X Factor but she decided to quit while she was ahead.
"There was no guarantee she would have been doing a second series. It made sense to make a clean break and concentrate on America.
"There’s no bad blood between her and the X Factor lot but they didn’t exactly get on like a house on fire."
The former Top Of The Pops presenter has already landed her first job in the US, co-presenting the NBC reality show Last Comic Standing.