The 21-year-old's dad Eustace was sentenced to nine years for GBH for attacking Mandy Thomas in 2003, reports The Sun.
The incident occurred at their family home in Bath, Somerset when Jahmene was 11 years old.
Eustace Douglas later admitted to raping another woman and was sentenced to a further six years.
Jahmene and his mother have had no contact with Eustace since he was jailed and are reportedly not sure if he is still in prison or not.
A friend of the singer said: "His dad was a nasty piece of work and terrorised the family. It's no wonder Jahmene is shy and has had confidence issues."
Jahmene revealed that he hoped to use his time on The X Factor to help campaign against domestic violence.
"Our struggles are what make us stronger and I hope I can become an inspiration to those still suffering," he said.
"I'm still here and happy. Don't let anyone take your happiness."
He added: "Only you know what's best for you and your family but speak out and reach out."
Jahmene recently revealed that he is still a virgin and wears a purity ring.
Photo gallery - X Factor second live show:
Copyright: Ken McKay/Thames/Rex Features