It was previously rumoured that he was considering a cameo on the reality show, as James is friends with some of the cast.
His sister Pippa Middleton has previously been linked with James Matthews, older brother of the show's star Spencer Matthews.
James said to the Daily Mail: "That's the first I've heard of it, and I've no idea where it's come from."
He dismissed the stories as "just a typical, speculated rumour" his family has become "so used to".
It was previously reported that Pippa, with whom James shares a flat, has landed a column for Waitrose's magazine.
View all of today's best celebrity pictures - Photo gallery: