McHale appeared on the late night talk show Conan on Thursday, where Conan O'Brien asked him about the Kardashians.
The actor revealed he is not liked by the reality star family.
He explained: "I bring up the thing that made Kim famous which is the sex tape with Ray J.
"Kim's mum would call E! much like an old lady neighbour going like, 'Your son's baseball landed in my window again', and they don't like it when we bring it up.
"I don't know why but we just say sex tape with Ray J all the time. And she calls and it's bad."
When O'Brien asked if it's because the Kardashians want people to forget about the tape, McHale quipped: "I'm promoting it. They make so much money off of it."
In August, McHale also said that Kardashian jokes are not allowed on The Soup, especially about the sex tape, because "the network flips out".
Although McHale pokes fun of reality stars in his E! programme, he revealed that he isn't a fan of reality show like Big Brother because they expose "human beings at their worst".
Watch the full interview of Joel McHale on Conan below: