The Take That singer recalled how one driver called him a "tw*t" due to his organised Diamond Jubilee Concert causing traffic delays.
He asked the driver if they could try a different route after they were stuck in a jam near Buckingham Palace earlier this week, reports The Sun.
The driver reportedly said: "Nah, some tw*t decided to build a stage here."
Barlow replied: "Yeah, I can't believe it", before keeping his head down for the rest of the trip.
Robbie Williams, Jessie J, Sir Paul McCartney, Stevie Wonder and Sir Elton John are among the long list of artists who will perform at Buckingham Palace.
The Diamond Jubilee Concert will air live tomorrow (June 4) from 7.30pm on BBC One.
The singer recently travelled around the Commonwealth to compose the Diamond Jubilee track 'Sing'.
Gary Barlow: On Her Majesty's Service will air tonight (June 3) at 7.30pm on BBC One.
Watch the video for Gary Barlow's 'Sing' below:
Photo gallery - Diamond Jubilee concert at Buckingham Palace:
Copyright: PA Images