European intelligence agencies were all aware of the type of covert surveillance undertaken by the US National Security Agency (NSA), a former state department official and current director of the Center for Strategic and International Studies (CSIS) has said.

"The European intelligence agencies knew what we were up to, so there was no surprise," said James Lewis, who led a discussion with senior US intelligence figures at the RSA conference, the large international security industry gathering here last week.

Regarding the outcry over NSA activities, leaked by whistleblower Edward Snowden, he said: "What surprised me was [that], I thought everyone knew this. Clearly this wasn't the case. [But] my conversations with people in other governments confirmed they knew we were doing this."

Also on the RSA panel, Richard Clarke, former White House counter-terrorism adviser and now chief executive of Good Harbor consulting – and a member of the advisory committee to President Obama on NSA reform – attributed a degree of "shocked, shocked" mock surprise to lawmakers and foreign governments.

French knew
"There were a lot of people – some of them French – who knew this was going on," he quipped. However, he said the general public in both the US and Europe would not have been aware of such tactics.

Retired former CIA and NSA director Gen Michael Hayden strongly defended the nature and scope of the NSA’s activities. “For people in my work, what constitutes consent of the government is how much of this do you have to know for things to be okay.”

He argued: “There is nothing unlawful going on here, based on the social contract we had coming out of the ’70s. The NSA was authorised by two presidents, approved by Congress by bipartisan majorities in both houses, and overseen by the FISA court. So, good to go”.

“This programme is solidly legal. What we’ve learned, what’s different, is it appears that’s no longer sufficient to define consent of the government.”

Some people, he says, “are saying, “Yeah, but you didn’t tell me. That, I think, is one of the really big tectonics we’re going to have to solve.”

Clarke said when he was in the White House, they used "the Washington Post test. If it leaks – or rather, when it leaks – can you and can the President explain it to the American people and the Congress in a way they find acceptable? I don't think US people have difficulty with their government doing some things in secret, but if it leaks, you ought to be able to justify it, and the majority of people and Congress should be able to say, "Yeah [it's okay]".

Disconnect
"Somewhere along the line, they forgot to do the test . . . there was a disconnect between what we needed to collect and what we did collect. Just because it is legal doesn't mean we should do it."

All identified a communications gap between policymakers and the intelligence community. “Policymakers didn’t take the time to explain what they wanted,” said Clarke, noting this had been true for several presidential administrations.

Lewis said he was worried “about the chilling effect of an intrusive oversight process” and how that might alter the ability to collect security information. Hayden complained that agencies found it difficult to satisfy public opinion. “We get criticised for not doing enough to make our citizenry feel safer and, as soon as people feel safe enough, we get criticised for doing too much.”

If lawmakers now move to curtail the NSA and other agencies, “we will be less aware, and your policymakers will have less situational evidence to base decisions on”, he added.

Lewis said other countries now had “a problem of trust” with the US, but Hayden countered that the US “is the most transparent intelligence community on the planet. European parliamentarians know more about American espionage than they do about their own espionage”.

The media weren’t reporting the issue very well, he said. People should “realise you’ve come into the movie late in the third reel and you think you know the butler did it, but it would have been good if you went back to the first and second reel to find why the NSA is the way it is”.

With today’s global communications systems, when millions of innocent people use them as well as lawbreakers and terrorists, “there’s no way for NSA to do what it does without mucking about in those channels”.

The agency didn’t really want to “touch your gmail account” but “had to make volume our friend. We had to start swimming and ride the wave of volume otherwise we’d be overwhelmed . . . in a universe of new communications, enemies and technology”, Hayden said.

Bulk data
However, Clarke cautioned that security agencies had claimed that bulk data from telephone calls, such as that collected from Verizon customers, was "allegedly very useful in stopping terrorism. But, as has been reported recently in newspapers, really only about 25 per cent of calls in the US are recorded in this programme. Which makes you wonder, if its so valuable, why they don't get the rest?".

For their report, his oversight committee went through 55 cases in which the NSA said such telephony metadata was important, but his opinion was the metadata collection programme didn’t help at all, and such claims were one reason more transparency on data gathering was needed.

“All their claims were bull. When you don’t have transparency, and don’t have people outside the system [in oversight], mistakes can happen.”

He said a large portion of Americans do not trust the NSA to oversee itself. The committee had recommended there should be third-party oversight of the NSA, people with high security clearance, capable of issuing reports.

Clarke said that, in the 1970s, it was clear that, eventually, technology would be available to support the creation of a police surveillance state, and once that happens “you can’t turn it off”.

However, “we don’t have a police surveillance state. Far from it, because we have lots of layers of oversight. But the technology is there, writ large, for a police surveillance state not just in the US, but in most of the modern world”.

That means the world needs to put up more roadblocks now.

“In the future, once you give up your civil liberties, you may never get them back.”

He added: “I think we’re in the process now of creating a new normal. One important thing Mike and I will agree on is, al- Qaeda has not gone away. It has metastasised. We have to have a strong intelligence capability to stop that. But the question is, can we have that with more transparency, and creating less collateral damage?”