Shirley MacLaine's debut in the hit period drama's third series opener was seen by 8.64m (34.6%) and 380k (2.4%) on ITV1 +1, around 200k lower than last autumn's second series premiere.
However, Downton inherited a smaller X Factor lead-in than it did in 2011, with the singing contest's final round of auditions drawing 9.33m (36.4%) in the 8pm hour and 397k (1.5%) on timeshift.
The Xtra Factor appealed to 1.14m (4.3%) at 9pm over on ITV2, adding 214k (1.1%) on timeshift.
Despite topping the night and setting a new series high, The X Factor's ITV1 ratings are down a sizeable 1.9m on last year's equivalent instalment.
Bradley Walsh gameshow The Chase made an encouraging start in the 7pm slot, entertaining 4.82m (22.7%) for its first celebrity special.
Overall, ITV1 reigned victorious in primetime with 31.1% (+1: 1%) over BBC One's 18.5%, Channel 4's 5.4% (+1: 1%), BBC Two's 5%, and Channel 5's 3.2% (+1: 0.4%).
Fake or Fortune? (3.79m/19.1%), Countryfile (5.82m/24.5%) and Inspector George Gently (4.33m/16.5%) aired on BBC One, while BBC Two screened China on Four Wheels (1.22m/4.7%) and Dragons' Den (1.75m/6.7%).
Channel 4's Big Fat Quiz of the 80s enthralled 1.69m (6.8%) between 9pm and 10.30pm, logging a further 286k (1.8%) on +1. Not a single programme topped a million viewers on Channel 5.
BARB ratings data supplied by Attentional