The 28-year-old, whose real name is Justin Edwards, alleged that the X Factor judge's supporters are intent on "destroying" him and his loved ones.
"I had millions of little 12-year-olds sending me death threats on Twitter," he told Loaded. "A lot of people made assumptions about what happened and why.
"Her fans have been destroying me. Some of them are like, 'Go and die'. It's been a hellish time having the world turn against you.
"It's not been nice for me and my family. I went on Twitter and wrote, 'All this fuss over something so little'."
Edwards also denied that he was responsible for leaking the tape, claiming that he had forgotten about the video shortly after shooting it with Tulisa backstage at an N-Dubz concert in 2010.
He said: "There were a couple of options of people it could have been and who could have put it out there."
Contostavlos claimed to have been so distraught at the release of the sex tape that she "slept on the bathroom floor for seven days", while her father said that the scandal "destroyed" their family.
However, the 'Young' singer has since attempted to play down the controversy, telling Alan Carr on an April edition of Chatty Man that the video "doesn't really class as a sex tape".
Watch the video for Tulisa Contostavlos's 'Young' below: