The ITV1 programme's recent third series sparked controversy after ten-year-old singers Natalie Okri and Hollie Steel broke down in tears on separate occasions in the live finals.
Reflecting on the row, show judge Holden told the Daily Mail: "Flippin' 'eck, when Hollie Steel broke down I legged it over to her to give her a hug and she was the one saying: 'Please let me go back on. Please give me a second chance'. She was the one demanding more.
"At the end of the day, how can you stop that? It's up to the parents to make sure that their children have realistic hopes. Simon [Cowell]'s mantra is that they should come in expecting to have a good time, but not to win."
Some commentators have claimed that a lower age limit should be introduced for the fourth series in response to the incidents.
However, Holden added: "If anything, I think there should be more children. I think we should have a special show just for the youngsters because there is phenomenal talent out there."
Dance troupe Diversity were voted as the competition's winners last month.