The television host told What's On TV that he had been a fan of the ITV period drama's first run.
"I loved the first series of Downton," he explained. "I mean, loved it. I got depressed during the final episode, knowing it was going to end."
However, he added: "I watched the first episode of the second series and it was terrible."
A recent report suggested that Downton Abbey's ratings are in decline, with viewers citing "excessive" advertising breaks.
However, an ITV spokesman later dismissed the claim and insisted that the channel is pleased with the show's performance.
"We're delighted that more viewers than ever are enjoying events from Downton with audience figures for the first three episodes this autumn up 20% on the first series," said the spokesman.
Downton Abbey airs on Sundays at 9pm on ITV1.
> Downton Abbey to lead ITV's Christmas Day schedule?
> Downton Abbey viewers crash ITV Player
Watch a trailer for the second series of Downton Abbey below: