The ITV show judge stated that too many contestants used Twitter during last year's competition, and wants the latest crop to focus on their singing, a source told Digital Spy.
"Last year's contestants spent too much time on Twitter when they should have been spending the time working on their songs and performances," Barlow is quoted as saying.
Barlow also explained that contestants struggle if they notice any negative comments online.
"They'd look at 100 comments and 99 would be positive but all they would think about was the one that was negative. It's not good for them.
"If they didn't see Twitter it would be much better for them. If I have my way I'll take their phones off them as soon as they go into the contestants house."
He added: "The X Factor is hard work, they need to put all their energies into their performances rather than worrying about who has said what about them on Twitter."
Gary Barlow was recently joined by Cheryl Cole for the Judges' Houses stage of the competition.
The X Factor continues on Saturday, September 15 at 8.10pm on ITV1.
Gary Barlow and Cheryl Cole at Judges' House:
Copyright: Ken McKay/Rex Features