The actor, who plays Donna Noble's grandfather in the current season of Doctor Who, said he was invited to audition for the title role in 1974.
He told The Sun: "When Jon Pertwee was leaving I went to be interviewed with a view to perhaps playing Doctor Who. I would have loved to have done it, I must say.
"We concluded the interview and I went home and didn't get the job, obviously. Tom Baker got it."
The 79-year-old is known to fans of the franchise for his role as Special Constable Tom Campbell in 1966 film Daleks - Invasion Earth 2150 AD.
He first appeared as newsagent Wilfred Mott in the 2007 Christmas special before becoming a cast regular alongside Catherine Tate in the fourth season.