Colton has been spending spare time with Johnson over recent weeks, but the Liverpool singer insisted that their relationship was purely a professional one at the moment.
"It's been said that it was a date - it wasn't a date," Colton told Digital Spy.
"He came to the show a few weeks ago. He's been working on his new album with some amazing people," said the ex-biscuit factory worker.
"They were the kind of people that I would want to work with if I did anything, so we met up and he showed me some of his tracks. They sound amazing. He was showing me some of them and hopefully if I leave the show I can work with some of the people that he's teamed up with."
Colton said that spending time with Johnson was also making him feel "at ease".
"I think Danyl's album next year is going to do really, really well and it's a nice peace of mind to know it's not all about being the winner," he said.
"So it helps a lot being around those types of people."
Johnson, whose first audition was described by Simon Cowell as "the best" he'd ever seen, finished fourth in the 2009 series of X Factor.
The X Factor continues tonight at 8.15pm on ITV1.