Totally annoyed!

[2careless]

I was taking photos of a very nice building (trying to do a 360 degree shot). I was positioned at the pavement outside this building and a couple of the security guards came out and told me I'm not allowed to take pictures of this building - 1. if I continue to stand there they'll just stand in front of me and block my views. 2. if I go opposite the road, then they'd call police.
Since I was alone and had rather expensive gear I didn't want them to be spitted on (potentially), I packed up and left.

I'm under the belief that if I was standing at the pavement I'm in a public place and these people have no legal legs to stand on threatening me like that. However, since this is Australia, the law can be quite different to other jurisdictions.

Has anyone been through similar kind of trouble?

P.S. If they call the police, what would an officer do? Can I file a complaint as well?