Havers, who took part in reality series I'm A Celebrity... in 2010, will play aristocrat Lord Hepworth when the show returns later this year.
Sharon Small (Mistresses, The Inspector Lynley Mysteries) will also have a major role in the ITV1 hit, according to The Mirror.
Small will play Marigold Shore, a housemaid of Lady Rosamond (Samantha Bond).
A show rep said: "We're very excited that Nigel Havers and Sharon Small will be joining the cast later this year. They are looking forward to getting started on set."
Zoe Boyle, Cal Macaninch, Iain Glen, Amy Nuttall and Maria Doyle-Kennedy joined the second series cast earlier this year.
> 'Downton Abbey' Q&A with Dan Stevens, Laura Carmichael
> 'Downton Abbey' star: 'New series is even better'