The singer, who failed to reach the final 24 on Sunday's (September 23) show, claimed that producers got rid of him to cause controversy, adding that fellow contestant Rylan Clark "can't sing".
"The X Factor set me up for a fall. They shafted me by praising me so highly - then sending me home," Whelan told The Sun.
"I nailed all three of my performances and didn't get one negative comment from the judges. So I was in complete shock when they kicked me out.
"But The X Factor loves controversy. They obviously decided the backlash was worth getting rid of me for."
Jahmene Douglas, Jake Quickenden and Rylan Clark were among those who got through to Judges' Houses.
"Rylan has a great look and great personality - but the man can't sing," Whelan added. "I'm not sorry I entered, though. Despite everything, it has been a great platform for me. Maybe I'll beat the winner to the Christmas number one this year."
An X Factor spokesman said that the "standard is very high this year and not all acts can make it through to the next stage".
Watch Joseph Whelan's Boot Camp performance below: