According to The Sun, the show's bosses are worried that people are bored with the format and are struggling to find willing contestants.
The programme features two teams, led by Anton du Beke and Darren Gough, competing in a game of human Tetris, where they attempt avoid being knocked into a pool of water.
Vanessa Feltz, Bobby Davro, Phil Tuffnell, Andi Peters and Alex Wotherspoon have all taken part in the first series.
"There's a lot of love for Hole In The Wall," said a source. "It's a hilarious show and it's found a place in the nation's heart.
"However, there are only so many times you can watch Vanessa Feltz get dunked in a pool and laugh. It just makes you groan after a while."
On the subject of celebrity guests, the source added: "It's difficult to get even Z-list names. We don't want to end up with former Big Brother contestants."