Kartheiser said that people didn't usually recognise him in the street, but that people automatically feel like they are not supposed to like him because of his role in the ad agency drama.
"The worst abuse I get is from people who don't know who I am. I was in a restaurant and the waitress didn't realise who I was. All she could remember is that she hated me," he told Shortlist.
"She'd spat in my food, but then was like, 'Wait! I'm sorry! You play Pete Campbell! I thought I knew you!'"
However, the actor claimed that he used to get a worse reaction when he was in sci-fi drama Angel.
"I think the kind of fan Mad Men attracts is quite intelligent. They get it, I'm not him. I was in a sci-fi show [Angel] for a couple of years and I got more hate for being in that - still do. Sci-fi fans take things a little too seriously," he said.
Kartheiser also played down suggestions that Man Men ever came close to ending during its 18-month hiatus when creator Matthew Weiner had contract wrangles with broadcasters and producers AMC and Lionsgate.
"This show has given too much and has been too important for anyone to let it just go," he said. "To end it now just couldn't happen, they shouldn't have let it come as close as it did."
Mad Men season five premieres on AMC on March 25 with two-hour episode 'A Little Kiss'.
It airs in the UK on Sky Atlantic on Tuesday, March 27.
> Mad Men Jon Hamm interview
> Mad Men exec: Matthew Weiner: 'I know how the show will end'