The War Horse star has amassed a dedicated fan base thanks to the success of his BBC drama Sherlock.
He told PA: "There is some obsessive behaviour, which is quite worrying. I worry for them, not for me."
The 36-year-old actor has two blockbuster movies in the pipeline - The Hobbit and the new Star Trek sequel - and appears in BBC Two drama Parade's End.
"I'm still very sensitive and wary of people recognising me," he added. "The only thing that really annoys me is people trying to surreptitiously take a photo on their phone without asking.
"I feel it's cowardly and a bit pathetic. Just ask me if you really want me to have a photograph with you."
Cumberbatch recently came under fire for his criticism of Downton Abbey.