The BBC issued an apology to Eduardo Medina-Mora Icaza earlier this week following jokes the show's presenters made about Mexicans and their culture.
Writing in The Observer, Coogan said: "I normally remain below the parapet when these frenetic arguments about comedy and taste break out. But this time, I've had enough of the regular defence you tend to hear - the tired line that it's 'just a laugh', a bit of 'harmless fun'.
Coogan went on to add that his own experience living in the US had taught him that Mexicans "work themselves to the bone".
He continued: "The Lads have this strange notion that if they are being offensive it bestows on them a kind of anti-establishment aura of coolness; in fact, like their leather jackets and jeans, it is uber-conservative... Big viewing figures don't give you impunity - they carry responsibility."
The comedian has appeared on Top Gear three times and acknowledged in his article that he was a fan of the show.