The actor is said to be taking part in the first episode of the new series, due to air next year, but will then depart the costume drama.
[Dan Steven's Matthew Crawley marrying Michelle Dockery's Lady Mary]
Although the reports have not officially been confirmed by ITV, a source from the show's production team told the Sunday Express: "Dan Stevens isn't returning to the series. He will probably do the first episode of the fourth series, but that will be it."
On Thursday (November 23), ITV confirmed that Downton Abbey has been picked up for a fourth series and a Christmas special in 2013.
However, Stevens will be appearing in this year's two-hour Christmas episode, which sees the Grantham family travel to Scotland, leaving the majority of the servants behind.
The 30-year-old actor is currently in the US performing in Broadway show The Heiress with Jessica Chastain.
Photo gallery - Downton Abbey Series Three finale: