The singer is said to have been rehearsing for the show, which would have seen him cover David Bowie's 'Life on Mars?' in addition to performing several of his own hits for the August 12 event.
However, he has decided to quit the production, as his wife Ayda Field is due to give birth around that time, reports The Mirror.
"Robbie quit a few weeks ago. He had been lined up to perform solo material and it was hoped he would sing with Take That," a source said.
"However, just as rehearsals were kicking off, he said there was an unfortunate clash of dates. It meant organisers had to adjust the entire schedule, because Robbie is an extremely high-profile performer, and had been given such an important role in the finale.
"A lot of organisers are now very unhappy. It's such a huge moment for British music and they thought they had the perfect line-up.
"Obviously, Robbie has to put his family commitments first. It's just a shame he didn't realise earlier."
A spokesman for Williams added: "He was asked to do the closing ceremony but due to the imminent birth of his baby, he couldn't take part."
Take That, George Michael and The Who are among the acts lined up for the closing ceremony.
Williams recently stated that he hopes his daughter will be born in the UK.
Watch Robbie Williams opening The Queen's Diamond Jubilee Concert below: