The This Morning presenter said he fights hard to keep the boys - Bobby, 9, and Freddie, 8 - out of the media and has a privacy ban on them until they are 16.
Brazier told The Sun: "They get mad at me sometimes. I took them to an event at the Science Museum recently and they saw me having my picture taken and they said, 'Why can't we get in the picture with you? Mummy used to let us'.
"You can't explain to them why it's good for them. Unfortunately, they get upset with me."
Jade Goody, the boys' mother, died of cervical cancer in 2009, and their lives have been kept private ever since.
Brazier admitted: "I probably could have been a millionaire by now if I'd gone down a different path.
"But it's not the way to find financial success and I want my boys to have a normal childhood."
The presenter also said that social media sites make it hard to keep the boys out of the spotlight.
He added: "I would love to share pictures of the boys and their progress on Twitter - everyone else shares pictures of their family - but I can't. But there will be a point when I have trouble which is probably not too far away when the boys want to go on it.
"They're already computer-savvy and trying to register for Facebook and Twitter and when they do go on them I will have to demand a privacy setting so only their friends can see their profiles.
"But who's to say they might turn around and say, 'No, I have a right to have a profile picture' - particularly when they are the only kid in the class without one. If they do, everyone will see an updated version of what no-one has seen for four or five years.
"They're big now and there's a huge difference in them. I just need to keep an eye on that and see how that one develops.
"There will probably be infringements on the privacy ban - but it may be more on their part, putting things out themselves.
"I don't know how I will control that - or even if I should control it."
Jeff Brazier is currently supporting the launch of Brewers Fayre's Local Heroes campaign and presents a daily I'm a Celebrity report on This Morning.