Last week, the girl band were chosen by mentor Tulisa Contostavlos to progress to the live shows, which begin this Saturday (October 8).
Chief executive of the music organisation Mark Davyd told The Mirror that he has enlisted a legal company to write to the show's producers.
Davyd said: "The first reply from X Factor was basically, 'We don't care'. Maybe I'm naive but I work in an area where I expect people to do the right thing.
"No-one would call a band Nordoff Robbins or Shelter. Now I accept that we are not as big as those organisations but we put on concerts and put out CDs and it's just not appropriate.
"We've started to have to pay out money to protect the integrity of the charity, which I just think is ridiculous."
However, an ITV spokeswoman stated: "The charity in question have trademarked the name 'Rhythmix' for educational purposes only, which is not in conflict with the band or the show."
> X Factor first theme week revealed, Cee Lo Green to perform
> The X Factor £6 million mansion: A house tour - In pictures