The 41-year-old judge told the Daily Star Sunday that last week's controversial decision to send the elimination vote to deadlock left him close to quitting the show.
"This show gets to me because it follows me round all week," he said. "And we are in it from now until Christmas. I hate it but I love it.
"Straight away after what happened last week I thought I don't want to do this anymore.
"I just thought 'this isn't the way I imagined it to be. After what I've just seen, the singer who did the worst has stayed."
Viewers saw Barlow storm off stage after Rylan Clark moved forward in the competition at the expense of his own act Carolynne Poole. The judge revealed that he left the set as he didn't want the cameras to pick up his reaction to the decision.
"I was even more upset than I appeared on the show," he said. "I was trying to hide it because I knew the cameras were on me.
"I couldn't stand the thought of them taking shots of my face while Carolynne was going through that. I couldn't bear it, so I walked off. I was angry with Louis and I blame him for what happened because he made her wait and wait and wait."
He added: "He told her she was staying and then that she was going. Deadlock is a horrible thing for contestants anyway, so to string it out was just awful."
Despite their apparent on-air reconciliation, Barlow said that he is still angry with Walsh for damaging the reputation of the show.
"The only two things I care about are the contestants and the fans of the show and last weekend didn't do us any favours," he explained.
"People need to know that when you vote and spend your money it matters and it counts.
"I hate the idea of it being fixed. The fact Rylan was in the bottom two shows things are changing and people want to see talent.
"But the whole thing about someone coming up on to the judges' chairs... there should be no-one up on that stage with us [and] I've told them that. I do not want it to happen again. We've never had that before. I didn't see it last year."
Barlow added: "I think in the short-term the reputation of the show has definitely been damaged.
"I know what I'm told by the public, by cabbies and stuff like that and people weren't happy last weekend. They thought their vote hadn't counted but it does count.
"I haven't cleared the air with Louis but yes, he has gone down in my estimation.
"If Simon tried to tell me who to vote for I would say 'No'. Nobody tells me what to do."
The X Factor continues tonight at 8pm on ITV1.
Photo gallery - X Factor second live show:
Copyright: Ken McKay/Thames/Rex Features