The comedian, who this week released new DVD Dandelion Mind, replaced Sean Hughes on the music panel quiz in 2002 but left the show himself in 2008.
Asked how he came to leave the programme, Bailey told DS: "To be honest, I was basically sidelined from the show. Buzzcocks normally records in October and the BBC moved the recordings forward two months and so it clashed with a lot of stuff I'd already booked.
"I'd had a tour of Australia, an orchestra tour and a West End run. They said, 'Could you just cancel them?' I thought, how utterly contemptuous do the BBC treat their artists? Thinking I'm sitting at home waiting for the phone to ring. You've got to get on with things.
"How rude are you to just assume I'm going to knock these things on the head? I said, 'I can't do it - I can't cancel these shows'. They said, 'How many can you do?', and I could do 7 or 8 out of 12 or something and they said, 'Well that's not enough', and I said, 'Well there you go'."
Bailey's former Buzzcocks colleague Simon Amstell recently said that he was not too mean on the show.
> Click here to read our full interview with Bill Bailey