The US comedian was filming a segment for a TV show in Kent while visiting the Countess Sondes when a group of balaclava-wearing guards approached her.
Rivers was enjoying a fishing trip on Swale Estuary in Kent - which is a known spot for illegal immigrants - while filming her reality show Joan Knows Best!.
Border Agency officials were allegedly alerted by a member of the public who had spotted a suspicious group of people on two fishing trawlers, reports the Daily Mail.
The guards interviewed Rivers, her daughter Melissa and Sondes before it was clear that they were not illegal immigrants.
Rivers revealed: "Even though we were innocent, I am so sorry they didn't arrest us. I love Englishmen and they were very cute."
Lady Sondes said: "Given the Lees Court Estate's close involvement with the Swale, it was a great comfort to know that the Border Agency acted in a proactive, efficient and friendly manner."
A spokesperson for the estate added: "It was all very dramatic. Joan was hilarious throughout the whole time she was with us.
"The Border Agency did not have the same sense of humour - they understandably took the situation very seriously.
"They were all dressed in black and were wearing balaclavas but you could still see their face - they were very efficient, not the sort of people you would want to mess with.
"But they were very professional and did a fantastic job - it's great to know people like that are protecting our shores."
The UK Border Agency yesterday confirmed that the incident took place on October 20, but dismissed reports that they were alerted by a tip-off.
A spokesperson said: "Our officers will investigate any potential immigration or customs offences and take swift and robust action to ensure our borders remain secure.
"On October 20, 2012, whilst on routine patrol, Her Majesty's Cutter Vigilant approached a fishing vessel in the Swale Estuary, Kent. Border Force officers spoke with the captain of the vessel to establish ownership and number of passengers.
"After a short discussion, Border Force officers were satisfied the passengers and crew did not present a risk to the border. The search of any vessel in UK territorial seas and internal waters plays an important role in protecting the border.
They added: "Presently there is no means of establishing, with any certainty, where a vessel has come from until Border Force officers have spoken to the captain, crew and in some cases boarded the vessel.
"In the vast majority of instances, officers will take no more than a few minutes to satisfy themselves as to whether a vessel presents a risk to the border."
Joan Rivers will perform at this year's Prince's Trust comedy gala We Are Most Amused at the Royal Albert Hall on November 28.