The X Factor judge managed Ward's career after the singer triumphed in the second series of the show.
"I thought Shayne had the X factor," Walsh told the Daily Mail. "I thought he ticked every single box. He had the looks, the talent, the ability, the attitude and the ambition. I believed in him 100%.
"I gave blood, sweat and tears in working with him - I absolutely did. Then there was no record, it all dried up."
He continued: "There was nothing I could do at all. My hands were tied. I only wish him the best because I think he's a great guy. But I reluctantly had to let him go."
Ward released his second album Breathless in 2007, but its follow-up has been pushed back several times, prompting fans to organise a protest last year.
However, the 25-year-old is still signed by record label Syco and latest reports suggest that his third album will be released in December.
Ward has also just signed a deal with a management company and his new manager Alan Edwards is confident that he can still be a success.
"I'm not surprised that Simon has not dropped Shayne because he really is a great singer," Edwards told the paper. "When he offered to play me some of his new tracks, he blew me away.
"I presumed it would be formulaic music, but it was very free feeling. I know Shayne's fans have missed him but music is art - it can't be rushed. I think that a bit of time between now and his last album could be an advantage - it puts a bit of distance between him and the show."
Edwards confirmed that the LP will have a Christmas release, which he claimed proves that the label is confident about the new material.
"It can be a double-edged sword going out at that time of year," he said. "You will be up against the latest Elton John Greatest Hits, but if you are confident in your music the returns can be fantastic - you can probably sell ten times more than at any other time of year."
He added: "The record label are confident in the album because Shayne is a proper singer and has a real future. He is serious about singing, not about being a celebrity. That's why you haven't seen him attending red carpet events and premieres - that is not what he is about. Shayne is a substantial artist. I think he could go on for years."
> Walsh: 'I still believe in Shayne Ward'