The One Direction star had complained on his Twitter page about "arrogant", "unprofessional" and "disrespectful" security at the iconic skyscraper after visiting with his family.
However, the Empire State Building's official Twitter account has now hit back at Payne.
"Bad behaviour not accepted at ESB," the person behind the account wrote, directly addressing Payne. "No, our security does not know who you are, and does not care."
Payne later appeared to backtrack on his criticism of the staff, tweeting: "Seems there was a bit of a mix up when I was at the empire state. Its all good. Sorry for confusion looking fwd to next timeeeee!"
Payne was in New York to perform with the rest of One Direction at Madison Square Garden on Monday (December 3).