The Avalon clip show's 11th series pulled in 4.38m (18.3%) at 7pm and 242k on ITV1 +1, its worst ever autumn premiere audience - down from last year's 5.7m and almost 2m lower than 2008's 6.2m.
But a repeat on ITV2 added an extra 339k (3.8%) at 11.45pm.
Rumours emerged a fortnight ago claiming the former stand-up comedian had quit the show, but he has insisted since that "no decision" has been made.
The X Factor's first live show picked up 10.44m (40.3%) between 7.30pm and 10pm (+1: 506k), and ITV2's Xtra Factor entertained 692k (3.5%) until 11pm (+1: 104k). A Celebrity Juice catchup followed Xtra Factor with 586k (4.7%) and 115k on ITV2 +1.
Meanwhile, The Jonathan Ross Show soared to 4.19m (21.2%) at 10pm on ITV1. A further 335k saw Ross interview Lady GaGa and Jamie Oliver on timeshift.
The National Lottery: Secret Fortune then mustered 3.68m (14.6%), and 4.02m (16.8%) watched Casualty.
Elsewhere, BBC Two's tribute to legendary comedy writer David Croft struck gold, as classic sitcoms 'Allo 'Allo! (7.30pm), Are You Being Served? (8pm) and Dad's Army (8.30pm) amused 2.3m (8.5%), 2.75m (10.3%) and 2.5m (9.5%).
Overall, ITV1 stormed primetime with 33.7% (+1: 1.5%), ahead of BBC One's 22.5%. BBC Two claimed 7.1% in third, followed by Channel 4 with 3.6% (+1: 0.8%), and Channel 5 with 2.9%.
Capping a winning night for the ITV network, drama channel ITV3 beat Channel 5 with a primetime average of 3.1%, after Foyle's War fetched 969k (4.3%).
Outside of primetime, England's loss to France in the Rugby World Cup scored 4.19m (41.7%) from 8am, grabbing a peak of 6.2m (52.4%) at 10am.
BARB ratings data supplied by Attentional