Chiles moved to the ITV breakfast programme along with his former One Show colleague Christine Bleakley last September, but the pair were fired just 15 months later following consistently low ratings.
Speaking after he and Bleakley hosted their final Daybreak episode earlier this month, Chiles confessed to being "dead wrong" for a daytime audience and noted that viewers had started to pick up on the "doubt" that he harboured.
"Looking back on it, am I right for breakfast TV? I thought I was dead right for it, and now I think I was probably dead wrong," he told the Daily Mail. "What housewife wants to look at me in the mornings? A lot of blame has to come to my door.
"At the end, we felt as if we were getting somewhere. That's the frustration of it. It did start to gnaw away. The game's up when you start to doubt yourself - and sometimes I did.
"I'm generally not happy in my skin. Everyone would say I didn't look comfortable, and the more they said it, the more uncomfortable and grumpy I got."
Chiles also addressed his somewhat fiery comments about ITV's handling of his Daybreak departure last month, explaining that he lashed out because he felt "humiliated" that the news had leaked.
"It got a bit messy in the end, as these things always do," he said. "Maybe if I'd gone in on all fours and begged, it might have made a difference. But probably not - the decision had been made at a high corporate level.
"Perhaps there was a conspiracy. I don't know… I was a bit humiliated. But it wasn't as if they'd leaked that I had six nipples - it was something that was true."
The 44-year-old added that still he does not regret his decision to defect from the BBC.
"I've learned so much about television in the last 18 months. I'm a better and much more experienced broadcaster than if I'd stayed at the BBC.
"With ITV you live and die by the sword, but I haven't got a problem with that. I'm battered, but not broken, and, ultimately, I still feel very lucky."