"It was a democracy, in the truest, most frustrating and most rewarding sense of the word," Downey Jr told Collider.
"Anybody could come in and say, 'You know, I'm just not cool with that'. We'd be like, 'Who's that?', and they'd say, 'Oh, I was just cleaning the trailers'. It was nuts," he added.
Downey Jr went on to speak about his relationship with Sherlock Holmes co-star Jude Law and director Guy Ritchie.
"Well, Jude and I are pretty close, and Guy and I are practically brothers, which makes things really interesting," he said. "There have been times when I've wanted to lop off his head with a machete, but it's just because I love him so much."
In October, it was reported that a third instalment in the Sherlock Holmes series is in the works.
Sherlock Holmes 2: A Game of Shadows will be released in US and UK cinemas on December 16.
Watch the trailer for Sherlock Holmes 2: A Game of Shadows below: