The music manager left the talent competition in 2007 after Dannii Minogue was recruited to the panel for its fourth series.
Explaining her motivation for dropping out, Osbourne confessed that she didn't want to give Simon Cowell the "satisfaction" of firing her for "someone young and perky-t*tted".
"It was a dumb move, but emotionally right," she told The Times. "It was a huge show; it paid so well, so much came from it. But I wasn't happy, so I walked away with dignity.
"Simon always ends up firing everyone. I didn't want to give him that satisfaction. He always wants someone young and perky-t*tted sat next to him."
Osbourne later assisted veteran judge Louis Walsh on the seventh series in 2010, but said that a permanent return was out of the question as Cowell wouldn't allow it to happen
Revealing that ITV bosses had previously called for her to come back, Osbourne continued: "Simon wouldn't let them and I don't give a s**t 'cos I can go, 'F**k you'.
"He's not a friend, but I don't not get on with him. We lead very separate lives: I'm family orientated, he's a mover and shaker."
Osbourne reignited her feud with Minogue last June, claiming that the Australian singer was "f**king thrown" off the show earlier in 2011.