Joachim Kempin, who worked at Microsoft between 1983 and 2002, overseeing the sale of Windows software to computer makers, said that if the firm is to "seriously get back in the game" it must look to introduce "a big change in management".
Kempin feels that this starts at the very top with Ballmer, a 30-year-plus Microsoft veteran who has acted as chief executive of the firm since 2000.
According to Kempin, Ballmer is too old at 56 to understand the 'Facebook generation'.
"Is he a great CEO? I don't think so. Microsoft's board is a lame duck board, has been forever. They hire people to help them administer the company, but not to lead the company. That's the problem," he told Reuters.
"They need somebody maybe 35-40 years old, a younger person who understands the Facebook Inc generation and this mobile community.
"They don't need this guy on stage with this fierce, aggressive look, announcing the next version of Windows and thinking he can score with that."
Whilst stating that he respects Ballmer, who was hired in 1980 by Microsoft co-founder Bill Gates, Kampin feels that the CEO may need to be part of the required management shakeup.
Kempin, who also had direct access to Gates during his time at Microsoft, has written a new book that is highly critical of his former employer.
He left Microsoft under a cloud in 2002 after some of the aggressive contracts he agreed with PC makers became part of the US government's notorious antitrust action against Microsoft in the late 1990s and early 2000s.
Published today (January 22), Resolve and Fortitude: Microsoft's "Secret Power Broker" Breaks His Silence reveals rare details of internal practices at the tech giant, and also gives a stinging criticism of Ballmer.
Speaking to Reuters, Kempin said that Ballmer has worked to force out any high-flying executives who could replace him as Microsoft chief executive.
He claims to have seen this first-hand with Richard Belluzzo, the executive who launched the original Xbox and was promoted to chief operating officer, only to last just 14 months in the post.
"He [Belluzzo] had no room to breathe on the top. When you work that directly with Ballmer and Ballmer believes 'maybe this guy could someday take over from me', my God, you will have less air to breathe, that's what it comes down to," he said.
Microsoft has not commented on the claims.
Ever since Windows unit chief Steve Sinofsky left Microsoft in November last year, there have been fresh questions over Ballmer's alleged treatment of potential successors.
Before Sinofsky, Windows and online head Kevin Johnson quit to join Juniper Networks; Office chief Stephen Elop joined Nokia as chief executive; and Microsoft's software guru Ray Ozzie quit to start his own project.
"Ozzie is a great software guy, he knew what he was doing. But when you see Steve [Ballmer] and him on stage where he [Ozzie] opposed Steve, it was Steve's way or the highway," said Kempin.
Kempin said that he challenged Ballmer two years ago over his management style, but had not seen any changes since then. He has also sent copies of his book to Ballmer and Gates, but not yet got a reply.
"Steve is a very good business guy, but make him a chief operating officer, not a CEO, and your business is going to go gang-busters," said Kempin.
"I respect that guy [Ballmer], but there are some limitations in what he can and can't do and maybe he hasn't realised them himself."
In his book, Kempin also reveals how Microsoft foresaw most of the major shifts in technology over the last decade, but botched its entry into tablets, smartphones and social media, allowing rivals such as Apple to gain ground.
"They missed all the opportunities they were talking about when I was still in the company," Kempin told Reuters. "Tablets, phones... we had a tablet going, we had tablet software when Windows XP came out, it was never followed up properly."
Several PC makers have publicly expressed unease at Microsoft's decision in 2012 to branch out into hardware with its Surface RT tablets. It will follow that up this year with Surface Pro versions, running the full Windows 8 software.
On this, Kempin commented: "Just think about the insult of Microsoft coming out with a tablet themselves, trying to mimic Apple, and now they are going to come out with a notebook on top of it."