Luigi and Angie Bellaviste are members of the Church of Satan.
Luigi told CBS News: "I feel like we're being treated unfairly because it's not a so-called mainstream religion."
Angie explained: "I know of many people who have the Virgin Mary and tons of Jesus memorabilia [like] 'I Love Jesus' - and what is the difference?"
The Bellaviste's front porch is filled with Satanist paraphernalia, including skulls, a black Christmas tree and the devil's numbers 666.
Although their neighbours find it a bit odd, they said: "It's still their property. It's still their house. They have a right to say whatever they have to say."
American musician and writer Anton Szandor LaVey founded the Church of Satan in 1966, and wrote the Satanic Bible.
Satanists do not worship the devil, rather they believe that human beings are animals and should pursue "physical, mental and emotional gratification".
Police confirmed to the news station that the crime will be filed as theft, and not as a religious attack.
Watch CBS News interview the suburban Satanists and their neighbours below:
> Couple banned from giving child 'demonic' 'Vampire Diaries' name
> Devil logo sparks Man Utd shirt ban