The Pointless presenter was considering joining the Channel 4 gameshow in 2008 as a replacement for Des O'Connor, but changed his mind after the executive's angry reaction to the move.
"She hit the roof when she heard that it was in the offing," The Mirror quotes him as saying. "You have to be very careful about those sorts of things. You think you're a free agent but you're not.
"I knew she was somebody who held power over a lot of key decisions on things I held dear and that if I did it, it would really p*ss her off. The ball was in my court, but I could sense that it mightn't be the best idea in the world."
Armstrong added that he was planning to start a quieter life with his family by taking the lucrative Countdown job.
"It had appealed to me because I didn't have any other work lined up," he said. "And, at that point in my life, it was a footballer's salary - well, almost. A second division footballer. I was in my 30s and started thinking we could move out of London and settle down."
Jeff Stelling was named as Countdown presenter in time for its 2009 relaunch, which also saw Rachel Riley replace Carol Vorderman.
Nick Hewer of The Apprentice took over from Stelling at the beginning of this year.
Hunt left her position at the BBC to become Channel 4's chief creative officer in September 2010.
> 'Countdown': 'Apprentice' Nick Hewer, Margaret Mountford reunite - video