The Paperboy actress, who starred in von Trier's 2003 drama Dogville, will no longer be a part of the film, says The Hollywood Reporter.
Kidman was allegedly going to do "a few days" of work on the film before leaving the project.
Charlotte Gainsbourg, Stellan Skarsgard, Shia LaBeouf, Christian Slater, Jamie Bell, Uma Thurman and Connie Nielsen feature in the cast for Nymphomaniac.
Jean-Marc Barr, Caroline Goodall, Kate Ashfield, Saskia Reeves and Omar Shargawi are new additions to the acting lineup.
Nymphomaniac centres on a woman (Gainsbourg) who recounts her sexual history to an older bachelor, played by Melancholia's Skarsgard, after he finds her unconscious and beaten up in an alleyway.
LaBeouf has previously claimed that he will have to shoot real sex scenes for the film, but producers subsequently said the film will simulate penetration with body doubles and visual effects.
Nymphomaniac will be released in two parts in 2013, with both 'hardcore' and 'softcore' versions.
Photo gallery - Top 10 movie sex scenes: