The rock singer and dad-of-one was one of the most popular auditionees on this year's series, but he wasn't selected for Judges' Houses at the weekend.
Jake Quickenden, James Arthur, Jahmene Douglas, Nathan Fagan-Gayle, Adam Burridge and Rylan Clark were the six singers put through ahead of Whelan.
Essex reality TV star Clark's selection has come under extra criticism because it is believed by some viewers he was chosen for non-musical reasons.
A Facebook page campaigning to get Whelan back on the ITV1 talent show has already collected 212,000 'Likes' after only a few days online.
Meanwhile, Digital Spy readers have responded angrily to the singer's omission.
One reader said: "By Sunday's show, Joseph was the only guy I wanted to go through. I liked his rock voice; much different sound to the other guys who pretty much all sound the same. I was absolutely flabbergasted that he didn't go through."
© ITV / Thames TV/Ken McKay
© ITV / Thames/Syco
Nicole Cross said: "Joseph Whelan should have gone through. I can't believe that idiot from TOWIE has a place over Joseph. Why should I be surprised, typical X Factor fodder. Not watching again in protest."
Charissa Ryan said: "Think it is a disgrace they chose Rylan over Joseph Whelan. I actually stopped watching after this and really won't be watching again."
Clark has responded to criticism of his selection on Twitter, telling critics to "go and have a w**k".
Nicole Scherzinger will select her top three Boys to take to the Live Shows this weekend.
X Factor Judges' Houses - photo gallery:
Copyright: ITV Thames/Syco
