Executives are said to be concerned after Brian Conley became the fourth celebrity in as many years to quit the show ahead of the eliminations, reports The Sun.
Comedian Conley was forced to leave the jungle after 10 days, after reportedly collapsing. He has since been discharged from hospital.
Celebrities are put through a series of tests by a team of experts to make sure that they can handle being in the Australian jungle.
Dr Sandra Scott - who is known for her work in the early series of Channel 4's Big Brother - is said to be in charge of the tests.
"We have to ask some serious questions about the decision to approve Brian as a contestant," an ITV insider said.
"Crucial warning signs may have been missed indicating he was not as well-equipped to deal with the situation as some people thought."
However, a spokesperson for ITV denied that any crisis talks were taking place, saying: "The safety and welfare of contestants is of paramount importance.
"All are assessed by independent medical experts before contracts are signed, and again before entering the jungle."
Conley's wife has also criticised producers for their handling of the comedian's health.
Nadine Dorries became the first person to be voted out of this year's I'm a Celebrity last night (November 21).