The Babyshambles singer, who dated Moss on-and-off between 2005 and 2007, said he had wondered whether the supermodel is "still in love with" him, but is sure she "hates [his] guts".
Speaking about their relationship at the Cannes Film Festival, where he is debuting his first movie Confession of a Child of the Century, Doherty revealed: "She was just a girl that I met that I really, really liked.
"The reason that it didn't work was because of all the other things. It really annoyed me that people in my life couldn't accept it in their heads that I was with her."
Doherty stated again that he has regrets about the end of their relationship and wonders whether he and Moss, who married Jamie Hince last July, could have worked out.
"I think, 'You know what, in a different world maybe we could [be together]'," The Mirror quotes him as saying. "It's all in her court. She wants to get drunk and call me and talk s**t. That's fine but I wouldn't be able to do that to her. Anyway, she's married.
"I've listened to her [on voicemail]. A couple of times she's got drunk and called me and been a bit silly.
"But she's a happily-married woman, why does she want to speak to me? Maybe she's still in love with me. No, she's not. She hates my guts anyway."
Doherty was reportedly spotted buying antique crack pipes with a Moss-lookalike back in January.
Watch the music video for Pete Doherty's 'The Last of the English Roses' below: