The British actor, who played Kumar's authoritarian brother in Patiala House, was attacked for his role as Jassi Kahlon.
"I was shocked at first but then realised how lucky - or unlucky in this case - I am to get recognition two years after the film has released. Just goes to show the power of cinema," Kirmani told Digital Spy.
Kirmani said that he was recognised by a cashier when he went to get a passport renewal form at a post office.
The cashier told Kirmani that although Patiala House was her favourite film, he played "a horrible brother" to Kumar and should come back later to pick up the document.
The 2011 Nikhil Advani project was largely shot in the UK at locations in London and Nottingham.
> What is Akshay Kumar's best film? Vote in our poll
Watch the trailer for Kumar's Special Chabbis below: