The About Time star was an early favourite to play the Time Lord when the BBC revived Who in 2005, prior to Christopher Eccleston's casting.
"I won't tell you when, because the rule is that you are not allowed to say you turned that job down because it's disrespectful to whoever did it," Nighy told the Daily Express.
"I will say that I was approached. But I didn't want to be the Doctor. No disrespect to Doctor Who or anything, I just think that it comes with too much baggage."
Nighy made an uncredited guest appearance on Doctor Who in 2010, playing art expert Dr Black in the episode 'Vincent and the Doctor' - written by About Time's Richard Curtis.
Peter Capaldi was recently unveiled as the 12th actor to be cast as Doctor Who's lead, with Nighy calling the Thick of It star a "brilliant, inspired choice" for the role.
Doctor Who will return to BBC One on Saturday, November 23 for its 50th anniversary special.
> Neil Gaiman on new Who star: Lack of black Doctor missed opportunity
Watch Digital Spy's Geek TV special about Peter Capaldi and the 12th Doctor below: