The Irish music manager also branded the 'Young' singer, who joined the show last year, as "rough around the edges", and accused her of thinking she is "a bigger star than she actually is".
"She is a bit rough around the edges and when they announced she was on the panel I was expecting a chav in a tracksuit - which is what we got," Walsh told the Radio Times.
"But she's a lot of fun... She thinks she's a bigger star than she actually is though."
Walsh's comments come hours after the first official pictures of him, Contostavlos, Gary Barlow and new judge Nicole Scherzinger were unveiled for the ninth series of The X Factor.
It was recently claimed that a contestant on the upcoming series toured with Contostavlos's former band N-Dubz prior to entering the show.
The X Factor returns Saturday (August 19) at 8pm on ITV1.
Watch the new X Factor trailer below: