The 31-year-old socialite sneaked out of Prime 112 restaurant in Miami on Sunday (September 30) to dodge the R&B singer, whose sex tape helped launch her career when it was leaked in 2007.
"Kim and Kris Jenner posed for pictures in front of the restaurant to allow Kourtney and son Mason to sneak out the side entrance unnoticed," a source told the New York Post.
"[By just] a few moments, Kim missed Ray J, who was seen entering the restaurant to meet a large group of friends including boxer Floyd Mayweather."
Kardashian's sisters Kourtney and Khloe - in addition to family members Kris, Bruce and Kylie Jenner - met for a family dinner at the steak restaurant to commemorate the nine-year anniversary of Robert Kardashian's death.
The reality star gave away their location by tweeting the group's whereabouts, causing a swarm of press to descend on the restaurant and apparently creating a distraction when she learnt Ray J would be arriving.
Another source told the paper claimed that Kardashian's relationship with Kanye West is "cooling off", because "he's getting tired of the constant press and promotion. He's not as tight with her as he was".
However, a third insider who is close to the Kardashians said: "The family did have dinner, but they never saw Ray J or Floyd. And there's no truth that Kanye and Kim are cooling off."