Host O'Leary, who has clashed on the show with his fellow countryman in the past, insisted that the cast "love him dearly".
© ITV / Thames/Syco
© ITV
Speaking on Daybreak, O'Leary explained: "It's always difficult with Louis. Do you know what - everything is good-natured with Louis, that's the one thing. He just doesn't care; that's the thing about him, you know."
The TV presenter added light-heartedly: "He's just like the old uncle in a dinner party, that's sat in the corner and just says - like uncle Jack - Father Jack.
"He just wakes up and says something kind of like, probably libellous and then goes back to sleep. We love him dearly."
O'Leary recently admitted that he has become tired of hearing the late Etta James's 'At Last', a song popular with contestants this year.
The X Factor 2012 launches tomorrow night (Saturday, August 18) at 8pm on ITV1.
Watch the show's series nine trailer below:
