I drove up to Sydney (bloody long drive) the other day and stopped by to check out the drain and orphanage in Goulburn. The orphanage is gone. It's been demolished.
That didn't stop a busy body old bloke ask me what I was doing as I was standing on the street out front looking at it. I told him I was standing on the street. He claimed to be a friend of the owner and said if I didn't leave he'd call the police. So I pulled out my phone and dialed 000, asked for the police and told them an old man was threatening and harassing me. He got back into his car and drove off.
Don't put up with s*** from busy bodies, call their bluff if you're on public property and just taking photos.