Cocozza departed the show on Tuesday after he broke a "golden rule" of the competition. He is believed to have been caught boasting about taking cocaine.
Barlow described the recent events as a "sad week" and claimed that nobody should feel "guilt-free" about what happened with the 18-year-old singer.
> 'X Factor' Frankie Cocozza 'thought drug use was no big deal'
> 'X Factor' Jonjo Kerr: 'Leave Frankie Cocozza alone'
Kelly Rowland was also critical of fans who booed Cocozza, claiming that she would not stand for any more booing from studio audiences.
All four judges were speaking at a conference to introduce their four returning acts - Amelia Lily, Jonjo Kerr, James Michael and 2 Shoes - who will compete in a public phone vote for the right to replace Cocozza in the talent show.
> 'X Factor' judges and wildcard acts: Press conference - Recap
> 'X Factor' phone vote to close during Saturday's show
Scroll through pictures from today's X Factor press conference below: