The Top Gear star, who is currently visiting Australia with co-host James May for the motoring show's live tour, was dining with Top Gear Australia presenter Shane Jacobson at Giuseppe Arnaldo and Sons when a fan tried to take some photos of them.
One onlooker told the Herald Sun that Clarkson was furious and grabbed the man's phone, slamming it onto the table, saying: "It was pretty loud. The whole restaurant heard the whole scene."
Another witness said that Clarkson had asked the fan to stop taking pictures several times, but he refused.
A spokesperson for Top Gear said that Clarkson's privacy was invaded but that the situation was resolved amicably.