Viewers expressed outrage after the controversial 19-year-old remained in the ITV competition over The Risk and Johnny Robinson this weekend.
© Ken McKay/TalkbackThames/Rex Features
© PA Images
O'Leary wrote on Twitter that fans had underestimated the amount of support for Cocozza and described speculation of vote-fixing as "a joke".
"Can tell most of you aren't happy, but if the acts don't get the votes they go out," he wrote. "It's that simple. Talk of a fix is a joke."
O'Leary was forced to deny similar allegations during last year's series of The X Factor after he was wrongly reported as admitting that producers had plotted to keep in unpopular contestant Katie Waissel.
He later said that the controversy over his comments made for "probably the hardest week of [his] life".
> X Factor Johnny Robinson defends 'young lad' Frankie Cocozza
> X Factor Frankie Cocozza: 'I probably should have gone in week one'
