According to TV Line, the music mogul explained that the 'Promise This' singer was given the opportunity to return to America but chose not to.
Asked to clarify the circumstances that led The X Factor to replace Cole, a veteran of the UK edition, with Scherzinger, Cowell replied: "We felt that she was going to be more comfortable doing the UK show than the American one [and] I thought it would be good for the show over there."
Cowell added that when Cole's return to the UK X Factor didn't pan out she was asked to return to the American show.
"She was offered a chance to come back to America, but for certain reasons decided not to. I guess that's what happens when you make reality TV - real things happen."
Cowell also dismissed claims that Cole was let go because she clashed with fellow judge Paula Abdul.
He said: "If it was a question of not getting along with Paula, I wouldn't be on the show!"
Cowell also commented that although Mariah Carey was unable to have a judging role on the talent contest because of her pregnancy, he confirmed that there's a section of the X Factor season that he hopes she will get involved with.
He joked: "She selfishly got pregnant, which is why she didn't end up as a judge."
The X Factor USA will premiere with audition episodes on Wednesday, September 21 and Thursday, September 22 on Fox.
Watch a promo for The X Factor USA below: