Carey has since taken up a position on the American Idol panel, but apparently initially expressed interest in the British music mogul's hit show.
Cowell said he did not think the singer would be right for his show as he believed she would struggle to be blunt enough with the contestants.
The 52-year-old told InTouch magazine: "Mariah called me about X Factor and I met with her but my main concern was that she might have found it harder on The X Factor than on Idol.
"Idol is a more gentle show and Mariah is not someone who wants to say no to people."
Cowell also revealed that he decided to quit American Idol because he had got "bored".
He explained: "I was bored on the show. I didn't think it was fair to the network that they were paying me to be bored.
"Over time on The X Factor the weak will go and the strong will survive. I believe with all my heart that The X Factor is the best show."
Photo gallery - The X Factor USA episode 2:
Copyright: Fox Ray Mickshaw