Baker, who took over the role of the Time Lord from Pertwee in 1974, also said with tongue-in-cheek that the late star was obsessed with paychecks, reports The Mirror.
"Jon Pertwee was an insufferable know-all," he told Doctor Who Magazine. "Jon found it physically impossible to buy a drink."
Baker went on to joke: "He liked the idea of big sums of money for voiceovers, so I would say in Jon's earshot that someone had offered me £15,000 for a voiceover, but I turned it down because it was going to take a whole hour.
"This wasn't true, but I could hear Jon's heart pounding. In fact, he died of a heart attack shortly after that. I think that's why."
Pertwee died in 1996 at the age of 76.