When Walsh has mentored the Overs, he has brought acts such as Wagner, Johnny Robinson, Kitty Brucknell and Tesco Mary to the Live Shows. Speaking earlier this week, the Take That star claimed that he wanted to bring some seriousness and credibility to the Overs this year.
Commenting on Barlow's remark, Walsh said: "Gary is new to this business, what does he know?
"I've been doing it for nine years. I know more about it than he does. I always work really hard with whoever I get."
Walsh also defended his track record on the ITV1 show, insisting that it didn't matter he hadn't won as a judge since series two with Shayne Ward.
"Well I have won again, because as far I'm concerned, JLS won," said Walsh.
"It's about what they do after the show is over. JLS had the best career.
"My three acts this year have the potential to get record deals. All three groups are here for it. It's a great showcase if you have something. And these lot do - they have a lot of it."
The X Factor Live Shows start this weekend at 8pm on ITV1.