The Tolkien Trust and original publisher HarperCollins are demanding £85 million in damages, saying the studio has not given them their share of the £3 billion the films have generated worldwide.
It is thought that two of Tolkien's elderly children and other family members are involved in the lawsuit, which was filed at the Los Angeles Superior Court.
Court documents claim that the Trust and a predecessor to HarperCollins signed the film rights to The Hobbit and The Lord of the Rings to the United Artists studio in 1969 in exchange for 7.5% of the gross receipts. The deal was inherited by New Line in 1988.
Accusing New Line of “unabashed and insatiable greed”, the plaintiffs are seeking compensatory damages, punitive damages and a declaration from the court that any further rights to the books, including The Hobbit, can be withdrawn.
The trustees' US counsel Bonnie Eskenazi said: "New Line has brought new meaning to the phrase 'creative accounting'. I cannot imagine how on earth New Line will argue to a jury that these films could gross literally billions of dollars, and yet the creator's heirs, who are entitled to a share of gross receipts, don't get a penny."