Julian White has claimed that he set up the group named Rewind last year, which included members Josh Cuthbert, Jamie 'JJ' Hamblett and Jaymi Hensley.
He has said that they had received interest from record companies and producers, but was fired by the group just before they found fame.
The group are then said to have hired Blair Dreeland - whose former girlfriend dated 2010 winner Matt Cardle - as their new manager.
Rewind later became Triple J, before X Factor producers later added George Shelley to make Union J, reports The Sun.
White said: "I made these lads and we signed an agreement for a contract. Next thing I know they've changed their name to Triple J and auditioned for X Factor with Blair as their manager.
Photo gallery - 'The X Factor' - Club Classics week results show:
Copyright: Ken McKay/Thames/Rex Features"They got all aggressive when I asked them why they'd gone. But I've consulted my lawyer and as soon as they start earning I'm coming after them for 5% of their earnings. I'm told I have a very good case."
However, a source close to the group said of the claims: "He managed them before George joined the group, so they're not even the same band when he managed them."
Current manager Blair Dreeland added: "Union J are confident that the ex-manager of Triple J has no contractual claims over the band."
One Direction recently gave their backing to the "talented" Union J and rival group District3.
Watch Union J singing 'When Love Takes Over' on The X Factor below: