The girl group admitted that they were told - along with the other contestants - by the star's management that they were not allowed to speak to her when she performed on the show back in 2011.
''We were on The X Factor with Rihanna and Lady GaGa," Jesy Nelson told We Love Pop. "Lady GaGa was amazing, but we didn't get to speak to Rihanna.''
Leigh-Anne Pinnock added: ''Her management wouldn't let anybody meet her, but she walked past me and said, 'Your hair is awesome!' This was when I had my head half-shaved, which I hated, so it cheered me up.''
Rihanna topped the UK album chart this weekend with her latest set Unapologetic, while Little Mix landed at number three with their debut DNA.
Watch the music video for Little Mix's 'DNA' below:
Photo gallery - X Factor latest results show: