Palestinian grocer Ayman Abu Aita brought the case against the pair over Cohen's controversial movie Bruno, reports TMZ.
The stars are arguing that the case cannot proceed in the US because two of the parties are not American.
In the film, a caption jokingly described Abu Aita as 'terrorist group leader, Al Aqsa Martyrs Brigade'.
Abu Aita claims that many viewers failed to understand the joke and now believe that he is a terrorist.
Letterman was included in the suit after airing the clip on his show.
The defendants' lawyers also claim that Palestine is not recognised as a foreign nation by the US.