The 19-year-old, who impressed the judges with a rendition of 'You'll Never Walk Alone' in his original audition, reportedly made the decision after his bosses at the Pakefield holiday camp said they could not guarantee him time off for the programme.
Bruton then competed against his fellow hopefuls to make it further in the competition. Viewers will see the remaining 150 acts whittled down to 12 over the next two weeks.
"I had to quit my job. It was Pontin's or The X Factor and I had to look at what, hopefully, was best for me in the long run," Scott told The Mirror. "I have put a lot at risk."
A Pontin's spokesman said: "Scott made the decision to leave because, understandably, he wanted to concentrate on The X Factor. He was told he would be welcome back at any time and that's still the case."
X Factor's first Boot Camp episode airs tonight at 7.45pm on ITV1.