The So Solid Crew star told Now that he got together with Cole after she split from her footballer husband Ashley and before she moved to Los Angeles.
On first being quizzed about the relationship, Harvey said: "Oh my God, oh my God! How the hell do you know that?"
He then added: "There was a relationship and attraction to each other's aura. I'm not going to sit here and say we went to this restaurant and that restaurant. But I'm sure people aren't stupid."
Cole replied on Twitter: "Was this 'relationship' happening in your head @harveyofficial?! Are you smoking something ?..
"I think I've met you once maybe twice at public events and With your wife !! #s**tjustgetsweirder"
Harvey responded: "Do you actually want me to tweet some of the messages you emailed me @cherylcole pipe down and stop playing the saint in front of your fans.
"I had your back @cherylcole untill you sent that s**t dissapointed in you if the public seen these emails you would end up with egg on your face."
He added: "And let's get this straight @cherylcole I don't need no hype of you! I'd rather go broke and work in sainsburys than talk about your personal s**t.
"This is not a twitter beef I still have massive respect for @cherylcole but you shouldn't of tweeted that knowing we both have personal messages to eachother #silenceinthecourt"
The rapper had continued in his Now interview: "We've known each other for years. But then [after Cheryl split with Ashley] we started confiding in each other.
"We had a lot of common ground and she was just a f**king nice girl. Oh, why did you have to bring her up, man? It brings back memories!
"The conversations we had... it was like two people who'd gone through so much and both respected each other because of it. I wouldn't have minded Cheryl being my wife!"
Harvey split from his then-wife Alesha Dixon in 2006 when he was found to be cheating with Dixon's friend and fellow singer Javine Hylton.
He later blamed Dixon's reluctance to have children for the break-up.
Hylton gave birth to the couple's daughter in 2008, but the couple split the following year amid rumours of his infidelity.
Dixon later said that she she forgave Harvey for cheating on her and had "moved on".