Downton Abbey's second series launched on September 18 with nearly 9 million viewers, but dropped to just over 8.7m for the third episode last Sunday.
The ratings decline is relatively minor, but it is in stark contrast to the show's steady rise over the last series and comes despite the programme enjoying recent awards success at the US Emmys.
The Daily Mail reports that viewers are upset over the amount of advertising breaks in the drama, particularly that the final part of one episode lasted only three minutes, including a highlights clip of next week's broadcast.
One viewer complained: "Nice of ITV to broadcast a bit of Downton Abbey in between the ad breaks."
Another added: "These bloody adverts drive you mad. Each one is over four minutes long and the last one was five minutes from the end."
Viewers have also complained that the launch episode of Downton Abbey was marred by a series of advertising idents run by insurance giant Aviva.
The new eight-part series of Downton Abbey features a 90-minute episode at the start and finish, with the middle episodes being 75 minutes. It is estimated that a 30-second ad slot during the programme would have cost up to £100,000.
An ITV spokesman denied that viewers have been put off by the adverts during Downton Abbey, pointing out that the show had a 36% share of the audience this Sunday, roughly the same as the previous two episodes.
He also claimed that the unexpected, record-breaking hot weather so far in October has led to TV viewing being down across all channels over the weekend.
"While the record breaking hot weather affected viewing to all television channels over the weekend, last Sunday's Downton Abbey was watched by exactly the same share of the overall television audience as the opening two episodes," the spokesman added.
"We're delighted that more viewers than ever are enjoying events from Downton with audience figures for the first three episodes this autumn up 20% on the first series."
> Downton Abbey viewers crash ITV Player
Watch the Downton Abbey cast discuss season two below: