Speaking to journalists in Australia, the Top Gear host reportedly compared the Prime Minister's handling of the economic crisis to that of Australian leader Kevin Rudd.
"[Rudd] genuinely looked terrified. The poor man, he's actually seen the books," he said. "[In the UK] we've got this one-eyed Scottish idiot.
"He keeps telling us everything's fine and he's saved the world and we know he's lying, but he's smooth at telling us."
The comments have sparked outrage in Brown's native Scotland, with Labour MP Gordon Banks slamming the remarks as "unforgivable".
Scottish Labour leader Iain Gray said: "Such a comment is really a reflection on Jeremy Clarkson and speaks for itself. Most people here are proud that the Prime Minister is a Scot and believe him to be the right person to get the UK through this global economic crisis."
The BBC and Downing Street have declined to comment.