Contostavlos, who broke up with her fellow N-Dubz rapper last week, mentored girl group Little Mix to victory in her debut series last year.
Speaking to Heat magazine, a source claimed: "He told her that he thinks she should quit The X Factor as she is better than that and is giving the band a bad name.
"It really cut through Tulisa and she started to cry. She now says that at that moment, she knew their relationship was dead."
The insider added: "She realises they are in a band together and have to sort out their problems but she isn't one to just get over things like this."
They are said to have split while on a romantic holiday in the Maldives.
> Tulisa Contostavlos avoids run-in with Fazer friend Chloe Green
> Tulisa Contostavlos debut LP thrown into chaos over Fazer split