Savage, who exited the BBC contest this weekend after performing a tribute to the late Wales manager, said that his critics should have more sympathy for Speed's loved ones.
"It was hard as you have to concentrate on dancing," he told The Sun. "But comments on Twitter like 'Robbie's only doing it for sympathy' were ridiculous, shocking.
"I don't want pity but I lost one of my closest friends. They should have sympathy with Gary's family. It's a huge loss to me but more importantly to his lovely wife, two great kids and adoring father, who travelled the world to watch us play.
"Gary was a family man, he was an icon. He had everything. I just hope now that everybody lets his family mourn and lets Gary rest in peace because that's what he deserves. It's the end for Gary, but his family have got to live through this for the rest of their lives."
Savage also said that he was thankful to have Speed's support throughout Strictly and would always remember his appearance at a live show several weeks ago.
"I didn't think he would come but he was there in the front row," he recalled. "He phoned me every single week.
"He loved the show. He was on the phone to me as soon as he heard I was doing it. He said, 'Go for it! Show people your true personality'. My lasting memory of Gary is high-fiving him in the front row at Strictly with a smile on his face."