Clark became the eighth act to leave the ITV show after falling into the bottom two with Union J.
In the sing-off, Clark performed a rendition of 'Wires' by Athlete, and Union J performed Snow Patrol's 'Run'.
Host Dermot O'Leary then turned to the judges for their votes. Only Clark's mentor Nicole Scherzinger opted to save him, while Louis Walsh, Tulisa Contostavlos and Gary Barlow saved Union J and sent him home.
Voting first, Scherzinger explained that Union J's performance was "very moving", but she had to send them home as she still believed in Clark. Union J's mentor Louis Walsh was next to vote. He described Clark as a "fantastic guy" and "brilliant fun", but explained that Union J won the sing-off.
Gary Barlow, who has been a harsh critic of Clark on this year's series, admitted that he was a "brilliant contestant".
He said: "I'm so glad because tonight that was your best vocal performance."
Contostavlos had the final vote and opted to send home Clark, saying she had to save the act who had more potential to sell records.
On ITV2's The Xtra Factor, Clark told hosts Caroline Flack and Olly Murs that he felt like a weight had been lifted off his shoulders.
He joked: "I feel like I've just been let out of prison."
Meanwhile, Barlow admitted that he and Clark have formed a bit of a "double act" on this year's show.
Clark also thanked Scherzinger for being his mentor, he said: "If it weren't for her I don't know what I'd be doing now, this woman has changed my life, so thank you."
Clark has been surrounded by controversy during the live section of the competition.
After Ella Henderson's exit last week, Barlow said that while Clark remained in the competition talented performers would leave the show, reminding viewers that the show should focus on vocals over performance.
His exit leaves four acts left on the show ahead of next week's semi-final. Christopher Maloney, Jahmene Douglas, James Arthur and Union J will compete for a place in the December 8 final in Manchester.
The X Factor continues next Saturday at 8pm on ITV1.
Watch Rylan Clark perform a medley of Spice Girls hits below:
Photo gallery - The X Factor ABBA and Motown week results: