Brazier successfully appealed for a ban on publishing photographs of Bobby, 8, and Freddie, 7, following the Big Brother star's death from cervical cancer in March 2009.
However, the 32-year-old confessed that he fears Goody's legacy could see them dragged away from the "stable, secure and fulfilling" environment he created for them.
"Initially, I was just grateful to take the boys out of the spotlight, but since then it means they have been able to enjoy a normal childhood," he wrote in The Mirror. "Aside from their obvious loss, their lives are stable, secure and fulfilling.
"The boys needed time to recover, grow and to find their feet after the rug had been pulled from beneath them... I need to give them a balanced perspective on fame because there's a chance that when they reach 18, there could still be some media interest. Nervous times ahead for a protective father.
"When it comes to kids, the vital work is done in their early years. If me and the boys maintain a close relationship, we will be able to communicate our way through every potential hazard, be it girlfriends, exams or large offers from glossy mags for photos."
Brazier added that he hopes the boys can weather the upcoming third anniversary of Goody's death.
"I have recently seen a dip in the boys' behaviour," he said. "Fred is scared to go to bed because of the bad dreams he's been having, and the other day Bobby shouted at me 'It's not easy not having a mum'. It's hard, but I know this phase will pass."
> Jeff Brazier: 'Fatherhood is my priority'