The 51-year-old comedian fell ill in the jungle and decided to quit the show following the advice of doctors.
A post titled 'So long, Brian' on the ITV show's official site read: "Brian Conley has left the jungle for medical reasons. We will miss him in camp and wish him well."
His wife Anne-Marie and daughters Amy, 15, and Lucy, 10, have flown to Brisbane to support him, according to The Sun.
No further details surrounding his illness have been revealed. Conley had been the bookies' second favourite to win I'm a Celebrity... Get Me Out of Here! behind EastEnders star Charlie Brooks.
Conley recently broke down in tears after clashing with boxer David Haye over the camp fire.