The two judges are guests on tonight's episode of Piers Morgan Tonight, were they will be seen discussing the British popstar's high-profile exit from the Fox show.
Cole filmed several dates of auditions alongside Cowell, Paula Abdul and LA Reid earlier this year. However, she was eventually replaced by Scherzinger, who had been co-hosting alongside Steve Jones.
Cowell said: "I was talking to someone about this, who was at the screening the other night, and they put it very well. She looked a bit bewildered. Not everyone is like you, Piers; you come to a country like America, it's a big deal, it takes a while to adjust."
He continued: "I took the view along with a few other people that she would have been more comfortable back in the UK. Because it was public, of course. I think if she had taken the UK gig, nobody would be feeling bad right now.
"That's what you have to do when you produce - you have to make decisions in the moment, and I take full responsibility for it."
Scherzinger added that she had not worked with Cole "that much" during the 28-year-old's time on the panel, but told Morgan that she thought she was a "gorgeous girl, very charming".
"For me it was an honour," she said of taking up the vacant position. "I'm not going to lie, I cried when Simon told me. I felt like no matter where he had me on the show, I was going to be of service and do the best that I could."
Cole will feature in the first hour of The X Factor USA, with Cowell having promised that the episode will tell the story of her exit "as it was".
Piers Morgan Tonight airs tonight at 9pm on CNN. Outside of America, the interview can be seen on CNN International at 8pm BST on Wednesday, September 21.
The X Factor USA launches with a two-hour, two-night premiere beginning tomorrow at 8pm on Fox.
> Simon Cowell: 'Cheryl Cole accepted X Factor return'
> Cheryl Cole on 'X Factor' USA was like Princess Diana out of England, says Simon Cowell
Watch Simon Cowell and Nicole Scherzinger discuss Cheryl Cole on Piers Morgan Tonight below: