Barlow accepted his own responsibility in the 18-year-old's exit from the show, following a drugs scandal earlier this week, but said that the media and vocal critics online and in the studio audience should not be feeling smug about Cocozza's exit.
"We've all encouraged Frankie to not only be good, but to be bad as well. We've enjoyed it, written about it. The show has benefited from the amount of publicity we've got out of it, but it's just gone too far out of control of the show to keep him here. I do feel a responsibility and I do feel sorry. I'm sad for him," he said.
"One of the worse things, and I'll always remember this about Frankie, he was constantly told he shouldn't be on the show.
> 'X Factor' Frankie Cocozza 'thought drug use was no big deal'
> 'X Factor' Jonjo Kerr: 'Leave Frankie Cocozza alone'
"He was told he wasn't good enough constantly. Whether the messages were online, on Twitter or in the papers, they all said, 'You're not good enough'.
"Then on Saturday night the audience booed him. I can see where his mind has gone in the last seven days and I don't feel good about it at all. Anyone who has contributed to that shouldn't feel good about it either."
All four X Factor judges were speaking at a conference to introduce their four returning acts - Amelia Lily, Jonjo Kerr, James Michael and 2 Shoes - who will compete in a public phone vote for the right to replace Cocozza in the talent show.
> 'X Factor' judges and wildcard acts: Press conference - Recap
> 'X Factor' phone vote to close during Saturday's show
Scroll through pictures from today's X Factor press conference below: