The mentors are said to have held meetings with their respective categories - the over 28s and the girls - and warned them not to get too close to the controversial contestant.
© WENN
Clark, 23, has been branded a bad influence by the two judges after he was spotted returning to the contestants' hotel drunk arm-in-arm with Tulisa's act Lucy Spraggan. The pair were later seen being kicked out of the building and are currently staying in a separate venue nearby.
An X Factor insider told The Mirror that Tulisa heavily criticised Spraggan for "not taking [the competition] seriously and throwing away her opportunity".
"Tulisa was disappointed with Lucy," the source continued. "She had her down as a serious musician and a real competitor for this year's title but now she thinks she's just here for a laugh.
"They really embarrassed themselves. Photos of them rolling around drunk in the street appeared and it really brings the show into disrepute."
© Rex Features
Spraggan later apologised on Twitter. "Me and Rylan Clark are partaking in a small vacation from the hotel for a few days," she posted, adding: "Our most sincere apologies for our #beerfear behaviour."
Barlow reportedly told his acts that Rylan has a "bad attitude" and shouldn't still be in the competition. "You'll ruin your chances if you're not careful," he said.
"Gary has made his feelings about Rylan quite clear," the source added. "But he's had another word with his own acts and warned them that there is no place for his messing around on the show."
The X Factor continues tonight at 8pm on ITV1. Contestants are due to perform songs that fit the theme 'Club Classics'.
Photo gallery - The X Factor live shows, week 2
Copyright: Ken McKay/Thames/Rex Features
