The four-time Oscar winner's use of recurring stars had led to stories in the press describing Diane Keaton, Mia Farrow, Penelope Cruz and Scarlett Johansson as being his inspirations.
Allen hit back at those suggestions while holding a press conference for his latest film To Rome with Love earlier in the week.
"When I walk through those red carpet things, the amount of times that I get asked, and have been asked, 'Is Scarlett Johansson your new muse?' 'Is Penelope Cruz your new muse?' [is frustrating]," he admitted.
The filmmaker added: "If I make one picture with somebody, they assume that I have a muse and that I want a muse, and that person wants to be my muse."
Allen added that another popular misconception is that he considers himself to be an intellectual.
"I'm not an intellectual at all. But intellectuals are amusing and so when I write, I write about them in comic dilemmas and in comic situations," the director explained.
"If I was pressed as an intellectual, I would be dead. But, as a writer, just making jokes about them, I can do that as something that interests me, because for whatever reason - whatever accident - I happen to find that part of the social milieu amusing and so it always seems to come out inadvertently in everything I do."
Allen also defended some members of the paparazzi this week when he said that many photographers can be "very nice" and "fun".
Alec Baldwin, Jesse Eisenberg, Ellen Page and Cruz star in Allen's latest movie To Rome with Love, now playing on limited release in the US.
> Woody Allen "wouldn't hesitate" to cast Lindsay Lohan in a film
Watch a trailer for To Rome with Love below: