Gary Barlow is the only judge officially confirmed for the new series of the ITV1 talent show so far. However, Tulisa Contostavlos and Louis Walsh have both strongly hinted that their returns are a done deal.
Commenting on rumours that Rowland's contract with the show is currently "up in the air", O'Leary told Digital Spy: "I would hope Kelly is up in the air literally. I hope she's in the air ready to do the show with us.
"It's a long old run, The X Factor. Sometimes with new judges, they can underestimate how tough it will be. And you can't chop and change all the time. We need continuity. You need to build these relationships as you go. Fingers crossed."
Joking about Louis Walsh's long stint on the show, the TV host joked: "Louis knows where the gold's buried. He's honestly protected by Keyser Söze, by God on high."
He added: "The great thing about Louis is that he's the most inconsistent human being in the world. You know where you are with him and he's brilliant.
"He's exactly what I need as a host. You know when all else fails you can have fun with him and bounce off him. He's incredibly thick skinned."
The X Factor returns in August on ITV1.