The Rising Damp actor appeared in many BBC productions in his career, including his iconic role as Reggie Perrin in The Rise and Fall of Reginald Perrin, before his death at the age of 57 in 1984.
The Sun reports that a man has alleged he was the victim of a sexual assault on the set of the Rossiter-starring TV play The Year of the Sex Olympics, while working on the production as an extra at the age of 18.
The accuser claimed that Rossiter was present for the attack and that he arrived after being "tipped off" that it was taking place. He was then alleged to have "watched with glee".
The man said that he attempted to report the abuse to a senior BBC figure working on the play but was ignored, then resorted to storming off the set with a female extra who had also reportedly been attacked.
The accuser, who contacted the newspaper after seeing the widespread coverage of the Jimmy Saville case, said: "It is important people know what a cesspit of depravity it was at the BBC. Staff at Television Centre assumed they could get away with anything.
"The fact the BBC didn't investigate the complaints by my woman colleague and me goes to show what the place was like back then.
"BBC staff got away with whatever they wanted. It's time now that these people face justice and I have given a full statement to police."
Other past celebrities who have been involved in the recent investigations into abuse at the BBC include disgraced singer Gary Glitter and comedian Freddie Starr.