The actor, who plays Bates in the ITV drama, bemoaned the Simon Cowell series for creating a generation of youngsters unaware of their limitations.
"I would have loved to have been a footballer like my great uncle Matt Busby, but I knew quite early on that I wasn't going to make the grade," he said to the Radio Times.
"Luckily I was told by the age of 13 that I wasn't good enough. That's not a bad thing. You see this X Factor generation of kids now who don't accept that they're not good enough."
Coyle also discussed the role that Downton Abbey has had on his career, revealing: "I was offered the chance to play a lot of moody men."
Coyle will next be seen in Starlings, which begins Sunday, May 13 at 8pm on Sky1.