In 2010, Bullock filed for divorce from James after he was caught having affairs with other women during their marriage.
James told Yahoo's OMG!: "I think I've paid the price and then some, you know? I don't know anybody anywhere that's had to endure what I had to endure for basically infidelity.
"It's just the nature of my life. People always will want me sainted one minute, and strung up the next."
One of his mistresses was Michelle McGee, who saw a photo of her wearing a Nazi military visor and swastika armband circulated in the media. Soon after, a photo of James miming an Adolf Hitler moustache in an SS visor was released.
"I stood up like a man and admitted my wrongs and what I did and all that did is make every s**t tabloid everywhere across the world come after me even harder," James said.
James also said that people were quick to judge him because of the way he looked.
"My older brother was gay, and looked just like me, and happened to die of HIV five years ago," he shared. "Nobody ever cared to ask about that.
"I think people just judge me by the way I look. That predates fame or any of that stuff. People have always been scared of me and always think I'm a bigot or a racist or a homophobe just by the way I look. I'm 6'3", and 215 pounds, [and a] tattooed-up white boy. It fits a criminal or whatever. They don't realise."
James recently landed a television special on Discovery Channel, called The Jesse James Project, which will air tonight (April 9) at 10/9c.
"If I could go back and change the way I did things, and not hurt people, and not do things the way I did, of course I would do it that way. I would never consciously bring pain to anyone, especially the people I love," he said. "It is something that I have to live with."