The bronze medal-winning swimmer was described by the comedian and former Mock The Week panelist as "possessing a dolphin's face" during the Olympics this summer.
Speaking to Digital Spy at the Collar & Coats Gala Ball for Battersea Dogs & Cats Home last week, Adlington admitted that she couldn't help but take Boyle's jibe "to heart", and said that such remarks were "bound to happen" once her profile raised.
"I think women especially are criticised more than men," she said. "It doesn't matter what men look like or what they dress in. I think a man can walk down [a red carpet] in pyjamas or a clown outfit and no-one would say anything.
"It's just part of the media life to be honest, you've just to get over it. I listen to my family and my friends."
She continued: "Obviously you do take it to heart, but you've just not got to get too down about it and just try and forget about it as much as you can."
Asked if she received an apology from Boyle, who also targeted her after Beijing 2008 by likening her to "someone looking at themselves in the back of a spoon", Adlington scoffed: "Oh no, but I don't expect him to. He's a comedian, it's his job."
The athlete previously described Boyle and other people who post abuse online as "quite sad".