The actress claimed earlier this week that Aubry is not fit to parent the child, leading him to strike back at Berry's allegations, which he insisted were untrue.
However, People reports that Aubry engaged in "vindictive" behaviour against Berry following their April 2010 split, even using racial slurs against her during heated moments.
"Once they broke up he became verbally threatening to Halle," his former business partner Stephanie Bibeau told the publication.
"He's a vindictive guy."
A source close to Berry added: "The only reason Gabriel is seeing Nahla right now is because the court hasn't made a decision yet.
"If it were up to Halle, he wouldn't be seeing her at all without some sort of supervision."