Security sensitivity in London

Is anyone aware of, or know of someone who would know, if there any guidelines/policies as to what represents a risk of security when photographing in London? Its OK to snap away at the 'Gherkin' on the opposite side of the road, but at what point (allowing for sinister 6'2" males loitering with rucksacks!) do security people tell you "No pictures!". I know in my case, its cos I'm wanting to shoot from underneath with a fisheye, but it either makes for a 'dive in and be ready to apologise' or an infringement on our photographic liberties. I also noticed the area around Whitehall, (I was photographing Millbrook Tower) sensitive too, but that is more understandable. I had thought of contacting Amateur Photography on this, they being London-based might be better informed. Does anyone else have any experiences/tips on this? (The east-European guard at the Gherkin stiffly presented me with a printed postcard with a photo and technical spec of his 'building'- as if I could use that to enter in bi-monthly competetions!)