Gingrich, who is currently campaigning for the Republican presidential nomination, recently accused the programme of being "anti-Christian bigotry" due to its satirical look at at Christian socialites in Texas.
In an interview on The View, Potts said: "I did feel that it was especially rich coming from him, that he called our little TV show un-Christian when he was carrying on an affair while he was impeaching the president for doing the same. It's like, you know, look in the mirror honey, I think our show might be for you!"
Potts is not the first cast member to respond to the politician's criticism. GCB star Denton Everett also defended the show against both Gingrich and New York City councilman Peter Vallone, who called for a boycott on the series.
"I know a lot of wind has been blown about the show," Everett said, "but it's like [series star] Kristin Chenoweth says, sometimes it's okay to laugh with it, not at it."
The programme, which debuted on March 4, also stars Chenoweth and Leslie Bibb as social rivals. It has earned steady ratings since its first broadcast.
GCB airs Sundays at 10/9c on ABC.
Watch Annie Potts on The View below: