The singer is said to have befriended the model after she admitted having a crush on him earlier this week.
Arthur had allegedly tried to keep his celebrity friendship a secret from his fellow finalists but was caught chatting to her at 4am on his computer, reports The Sun.
A show source said: "James came into the contest thinking he was this big ogre that girls would never fancy.
"But now he's been caught red-handed flirting with one of Britain's most famous glamour girls — and she has been coming on to him.
"It's not easy to see what they both have in common, apart from a love of tattoos, but they seem to be getting on like a house on fire."
The insider added: "He's been trying to keep it quiet but was busted and looked all red-faced."
James Arthur was previously linked with Take Me Out star Chanelle McCleary, after dumping his dancer girlfriend Lucie Poole.
The X Factor continues this Saturday (October 13) on ITV1.
X Factor's Top 10 Novelty Acts:
Copyright: Rex Features Ken McKay/TalkbackThames/Rex FeaturesNew! Digital Spy breakfast show - watch Daniel Sperling discuss the day's biggest stories including Rihanna's next album, Quentin Tarantino's Django Unchained and The X Factor's Rylan: