The Campaign actor said that Axel - his third child with wife Viveca Paulin - is being encouraged by his older brothers, Magnus, 8, and 5-year-old Mattias to insult different people.
While Ferrell is trying to discourage his behaviour, he admitted that he finds the joke very funny.
Speaking on The Late Show with David Letterman, he said: "The 2-year-old unfortunately is just wandering into vulgarity - he has two big brothers, it wasn't from me.
"I'm trying to prevent it. I'm doing everything humanly possible. I'm having a good laugh and it's out of my hands, I can't control it."
He continued: "He thinks it's hilarious - and so do his brothers - to walk up to anyone, male or female, and say, 'You have a little penis'. He'll get on a roll and say it over and over again."
Ferrell confessed that he is planning to use his child's insult to prank some famous friends.
He explained: "My brother-in-law wants me to have a big Hollywood party and invite loads of actors, then hopefully he'll walk up to a George Clooney type and say 'You have a little penis'. He thinks it would be really funny."
Watch the interview below: