Oliver, 19, passed away just two weeks ago following a battle with bone cancer - four years after their dad died of the same disease.
"I was at his house with the rest of our family when it happened. We were all with him. It was devastating. But it was good he had everybody who loved him around him. It made it easier that everyone was there and we all got to say our goodbyes," Jake told The Sun.
"He had been ill for a while but it was still a surprise. His breathing changed that morning. My mum said, 'Everyone needs to be here now'. It was quick. He just stopped breathing and that was it. It was just disbelief at the fact he had gone and everyone loved him so much.
"He was everyone's little hero. I don't think we will ever get fully over it. We all watched my audition on The X Factor together. Olly wasn't very well but we had a Chinese and he said how proud he was of me."
Oliver was diagnosed with osteosarcoma - the most common form of bone cancer - in early 2010 when he was 16. He proposed to his girlfriend Olivia, 19, in April last year after his condition deteriorated and the couple married that July.
Oliver was seen supporting Jake at his first audition, which aired earlier this month on ITV1.
The Sun made a donation to the Teenage Cancer Trust for Jake's story. Watch his X Factor audition below: