Sir Tom Jones said last month that the judges on ITV's singing competition are "cruel to people needlessly".
O'Leary has now said about the BBC programme: "Everyone who goes on those shows does so to be judged and maybe an even harsher critique is someone not turning around. It is what it is. You can't complain if you don't like what you hear. Well, you can and usually it's to me!"
However, he went on to reiterate that he likes the rival show, saying: "I'm enjoying it. It's good we've got competition and it will help us raise our game."
The Sun reports that the X Factor host also praised his show's "fun" factor and "ridiculous" contestants, commenting: "The most important thing is to have fun on the shows. And just because auditionees are a bit strange, it doesn't mean they are not good.
"I like the fact we are about the package. I like that we get acts like Johnny Robinson who look ridiculous, but it doesn't mean they can't sing."
O'Leary also spoke about the returning judges, saying: "Gary is confirmed. I'm happy he is back. I want Gary to embrace the madness more, though. I would like to see everyone back.
"Yeah, Tulisa looks to be guaranteed and Louis will be back."
It was reported today (April 27) that Kelly Rowland is unlikely to return as a judge.