The 81-year-old actor had signed up to provide the voice of Charlie but has now left the project, TV Guide reports.
In a statement, ABC and the show's production company Sony confirmed the news.
"Due to his scheduling conflicts, Robert Wagner will need to step away from his duties as Charlie on Charlie's Angels," they said.
The voice used in the show's pilot was reportedly temporary and casting is thought to be underway for a new actor to take on the role of Charlie.
Charlie's Angels, which stars Rachael Taylor, Minka Kelly and Annie Ilonzeh, will premiere on September 22 on ABC.
> 'Teen Mom' star to appear in Charlie's Angels?
Watch a trailer for Charlie's Angels below: