The actor, who is a longtime friend of the period drama's creator Julian Fellowes, spoke of his relief at being asked to play Lord Hepworth in last year's Christmas special.
"I've known Julian Fellowes all of my life so I was really hacked off that he didn't ask me to be in it in the first place, but now I've finally been asked to be in it, it's really nice," he told Lorraine.
Havers is currently starring in ITV1 soap Coronation Street as male escort Lewis Archer.
"Being in Coronation Street has been one of the highlights of my career," he said.
"It was a big honour when I was asked to go on. I said, 'God, I'd love to go on, I'll do it straight away'."
He added: "I was very nervous and I was amazed by the street because the houses are two-thirds of the actual height. I love doing it."
Havers recently said that he was unsure about the future of his character Lewis Archer in the soap.
Meanwhile, Downton Abbey executive producer Rebecca Eaton has revealed that a major character will die in the show's third series.