The 'Safe' singers recalled how the Irish music mogul brutally mocked their appearance, banned them from having relationships and eagerly axed original member Derrick Lacey.
"We handed him the CD with pictures of the six of us," bandmember Kian Egan told the Daily Mail. "He covered one of the guy's heads with his thumb and said, 'Looks much better, doesn't it?'
"[He] said, 'I'll manage you, but you'll have to do one thing... The big guy who's jumping around like a lunatic up there - get rid of him.'"
Explaining that the X Factor judge used to mock Nicky Byrne's horrible teeth and Shane Filan's weight, Egan added: "We lived in fear of Louis. He was like, 'You're done. I'm going to kick you out of the band. I'm going to put you on the next plane home. Only four people can fit in a taxi, not five.'"
Filan continued: "We were definitely scared of him. He was a very powerful man - only a tenth as powerful as he is now - but the best manager in the world for us."
However, the group insisted that they are pleased with their working relationship with Walsh and relish seeing him being playfully mocked on the X Factor panel.
"Oh we're all fond of Louis," Filan said. "Simon's now doing to him what he did to us, so he's gone and got everything done.
"His hair's browner, his teeth are whiter and he seems to be using moisturiser a bit more too."