The comic said at a Montclair Film Festival fundraiser over the weekend that he was initially unimpressed with The Daily Show staff when he replaced Craig Kilborn as host in 1999.
"What I did not realise is, a lot of the people who worked there were assholes," Stewart quipped, reports Third Beat.
He went on to say of his troubled first year on The Daily Show: "I had, before taking [the job], some conversations with the powers that be there about the direction I thought we could move the show... I wanted it to be satirical in the classic sense of the word, not the Spy magazine sense of the word where you just add adjectives like 'pepperpot'.
"I walk in the door, into a room with the writers and producers, and the first thing they say is, 'This isn't some MTV bulls**t'... And then I was told not to change the jokes or improvise. [I told my agent], 'Get me the f**k out of this'. These people are insane."
Stewart went on to reveal that it took him nearly three years to feel comfortable at The Daily Show, in which time the show experienced a substantial staff turnover.
He has remained on the Comedy Central series for the last 13 years and has interviewed luminaries such as US President Barack Obama and ex-British Prime Minister Tony Blair.
The Daily Show airs Monday to Thursday on Comedy Central at 11pm ET.