The 'Party In The USA' singer, who was linked to the show back in May, admitted that she didn't want to earn a reputation as a "dream crusher".
"I just don't know if I want to be some dream crusher," she said, according to The Sun.
"Would you really want to sit there as someone is belting their heart out - even if it is like hearing nails on a chalkboard - and then say to them, 'You suck'?
"I have no filter, so I would be scared I would say something and they would go home and say, 'Miley Cyrus ruined my dreams'. It seems harsh."
Britney Spears and Demi Lovato were eventually unveiled as the replacements for Paula Abdul and Nicole Scherzinger on The X Factor USA.
Photo gallery: The X Factor USA season two Judges' Houses:
Copyright: Fox Ray Mickshaw