I can see your point.
But some times if people are around it is actually worth introducing yourself before you go into the site and saying you're a photographer. But yeah, some people will just have a go at you anyway.
Years back (because I'm old) I was at OPEX 77 in Buffalo New York and there was a wonderful old church that we wanted to explore. When we got there there was a whole pile of people from the nursing home next door looking at us. So I walked over and said "Hi, we're not vandals, we're just a bunch of photographers, and we're here to photograph the church."
The Aussie accent went down well, as did the obvious number of cameras, and in we went down a grate into a basement in full view of these people and not a word of complaint was made. Some of the explorers even talked to the old folks about the history of the church, finding out it used to be a Polish church but due to "white flight" the church was abandoned as the Polish people moved out.
Hmm, maybe fake an accent and make like a tourist?
http://scotthaefner.com/photos/place/Buffalo/3070/