The businessman was due to take on the challenge with Ronan Keating, Jenny Frost, Pamela Stephenson, Jason Bradbury and Olympic medal-winning swimmer Steve Parry to raise £1 million for Cancer Research UK.
However, PA reports that he will no longer complete the swim after a fire destroyed Branson's Great House on Necker Island earlier this week.
Actress Kate Winslet was among 20 people who fled the fire, saving the life of Branson's mother along the way.
Meanwhile, Keating will still take part in the swim despite having an injured hamstring.
"I'm not a strong swimmer, but the motivation to learn and raise money for Cancer Research UK was enough for me to sign up," he said.
"The public's support is going to be essential in helping us achieve our goal of raising £1 million for Cancer Research UK's life-saving work."