Barlow and Clark have had numerous run-ins on the ITV1 talent show and the Take That star singled out Clark for criticism after he survived ahead of Carolynne Poole in week one.
Following yesterday's shock elimination of Ella Henderson, Barlow told the judges that they shouldn't be surprised anymore if good acts leave, when they have previously voted to keep Clark on the show.
"I'm more annoyed Rylan's still here. I said from day one, every week that Rylan's here, a good act will go home and here's the result tonight," said Barlow.
Watch Rylan's performance from Guilty Pleasures week:
Barlow also defended his final Overs act Christopher Maloney, who was booed by the studio audience on Sunday night.
"Christopher can actually sing," said Barlow.
"If there's people out there willing to spend money on phonecalls, I think they would buy his album. I'm sure there's people willing to spend money on [Christopher] albums."
X Factor Shock Exits - In Pictures: