Fern is currently halfway through a month-long trial run on the broadcaster, but executives are said to be concerned that the show has failed to attract over one million viewers - averaging just 970,000.
Earlier in the year, the channel was attracting up to 2.4 million viewers in its 5pm slot with Come Dine with Me and Coach Trip.
The Mirror reports that Fern was beaten by its rivals on all the main channels on Thursday. BBC1's Weakest Link got 1.8m viewers, Britain's Best Dish achieved 1.5m for ITV1, while BBC2's Escape to the Country attracted 1.3m viewers. Neighbours pulled in 1.2m viewers on Channel 5 from 5.30pm.
"It's fair to say the show is in jeopardy," a source said. "The figures are relatively low - they are certainly not as high as we hoped. But it's early days. We'll get to the end of the four-week trial and decide what's best for the channel and the show.
"Everyone here thinks Fern is brilliant so we can't understand why more of her loyal fans aren't watching. The good weather at the moment probably isn't helping."
Channel 4's chief creative officer Jay Hunt and head of daytime Helen Warner will decide the fate of the series at the end of the month. Options are said to include changing the format, moving it to an earlier slot or axing it completely.
Britton recently insisted that daytime television presenters are underestimated.