Barlow made the comment about his fellow panellist during a live show two weeks ago after she accused his act Christopher Maloney of "destroying" a series of '80s classics.
He later apologised to Contostavlos and gave her "the freedom to be as personal as you like" with him for the remainder of the show.
However, in an interview with the official X Factor website, Walsh claimed that the pair's relationship will "take time" to mend after Barlow's insult.
"They're pretending they're best friends but I know it's not 100% yet," he said. "It's going to take time for their friendship to go back to what it was.
"I won't be intervening. Tulisa is very much her own woman and doesn't need me to fight her battles."
In her own Q&A, Contostavlos claimed that she is "cool" with the rest of the panel, saying: "Louis loves a good gossip but not in a spiteful way.
"This may surprise people but he knows how to keep a secret."
Contostavlos recently admitted to Digital Spy that she is "paranoid" about her smoker's breath.
Watch Digital Spy chat to Tulisa Contostavlos below:
Photo gallery - X Factor results show 6:
Copyright: Ken McKay/Thames/Rex Features