The 33-year-old from Colchester, Essex has signed to present the next two series of the TV singing contest.
He said: "I was very flattered to be offered the job to host such a hugely successful show. I’m really looking forward to working with Simon and the team on the kind of Saturday night entertainment show I grew up watching."
O'Leary has previously presented Big Brother’s Little Brother, Big Brother’s Big Brain and the BBC’s lottery show 1 vs. 100.
The fourth series of The X Factor begins on ITV in the summer. Hopefuls will be split into one of four categories - 14– 24s Boys; 14 – 24s Girls; 25s and Overs and Groups - and there will be a fourth judge.