The Big Brother host informed police that the robbers, who stole his money and phone, held a knife to his throat.
Dowling told The Sun: "They emptied the contents of my wallet, went through my pockets and took my phone.
"I was terrified. I live in a lovely area, it was only nine at night and was only 30 seconds from my apartment."
Dowling, who resides in Lambeth, continued: "You don't expect it at that time of night, or on a quiet residential street.
"All I could think of was, 'Are they going to stab me?'. I felt physically sick when I got home."
Dowling added that passers-by on the other side of the road witnessed the incident, but were "unaware what was going on".
A Scotland Yard spokesman confirmed: "At around 9pm on Wednesday we were called to reports of a man in his 30s being robbed. No arrests have been made at present."
Dowling has fronted Big Brother since its move to Channel 5 in 2011, having appeared in the reality show as a contestant in 2001 and winning Ultimate Big Brother in 2010.