The Harry Potter actress is said to have "screamed in terror" after recognising the man from a previous encounter.
"The man managed to slip on to the set with some visitors and then approached her. She recognised him from hanging around outside her house and screamed," a source told The Sun.
"He took off into the woods and was chased by security and a couple of the tough stunt guys who also happen to be martial arts fighters.
"They caught up with him in the woods. Police were called and he was removed by security. It was really creepy. Emma was really shaken up and was given some time to compose herself."
Watson was shooting biblical epic Noah in Oyster Bay, Long Island. The film also stars Russell Crowe and Sir Anthony Hopkins and is scheduled to be released in spring 2014.
"It's been a really tough shoot for her. The movie has been gruelling anyway and this left a very sinister feeling hanging over the set," the source concluded.
The man was not arrested or charged but was warned about trespassing.
Watch the trailer for Emma Watson's latest film The Perks of Being a Wallflower below: