The extent of US data collection now suggests something like a mania in Washington, with reports of allies being bugged as well as enemies. But where does the expertise come from? When it comes to data collection, there is one digital industry that specializes in it. It is barely a decade old, but it has taken over almost every aspect of our lives - social media.
The border between protecting private data and collecting it is narrow, and, as it turns out, easily punctured. Last month, the New York Times revealed that in 2010, Max Kelly, the man in charge of protecting the personal data of Facebook's 1.1 billion users, left his job and moved into a post at another major US organization involved in data analysis: the National Security Agency, the branch of the Defense Department responsible for collecting and analyzing digital transmissions.
Kelly, the former chief security officer for the world's biggest social media platform, had moved to the spy agency just a few months after its PRISM program had begun to collect data from Facebook. Kelly's move - kept secret for three years - provides a stark illustration of the deep connections between NSA activities and social media.
"Silicon Valley has what the spy agency wants: vast amounts of private data and the most sophisticated software available to analyze it," the newspaper wrote. "The agency in turn is one of Silicon Valley's largest customers for what is known as data analytics, one of the valley's fastest-growing markets."
The New York Times also reported that to pursue its ends, the NSA has even turned into a venture capitalist investor - ploughing some of its estimated $8 - 10 billion (6 - 7.5 billion euro) budget into Silicon Valley start-ups.
'Google seeks defense contracts'
It's obvious that state intelligence organizations have always been interested in collecting data, and IT companies have long been selling their equipment and expertise to the US government, but the rising value of customer data for consumer marketing purposes means that data collection has been a key point of interest for commercial companies too. One side needs the intelligence, while the other wants profits. Coupled with exponential advances in data storage technology, these are perfect circumstances for mutually beneficial cooperation.
"US online marketers have created a pervasive and unaccountable personalized data collection and targeting apparatus - they really can gather and reach us nearly 24/7," said Jeffrey Chester, executive director of Washington-based digital rights NGO Center for Digital Democracy. "It's not really us providing the data voluntarily. The Prism debate should focus on the growth of the unaccountable commercial data empires and facilitating technologies."
The link between social media companies and government agencies is not news to Chester. "There's always been a revolving door between phone companies and government agencies," he told DW. "Companies like Google and Microsoft actively seek Defense Department contracts."
"I think it's very troubling that the NSA is recruiting someone who is so high up at Facebook," he added. "The NSA is simply replicating what social media surveillance is already doing. While Facebook and others claim social media is a tool for empowerment, it's also being widely used for wide-ranging political and corporate surveillance."
Size and complexity
The scale of the cooperation between the NSA and social media remains a mystery, but the technological capabilities of social networks are well-known. "They are tapping real-time geographic location and history," Chester said. "They have been working to expand their ways to mine information on their users and their activities. They have created so-called 'social media command centers' with which they analyze and monitor social media."
Simon Weiss, of the Berlin branch of Germany's Pirate Party that specializes in Internet freedom issues, says that there is as urgent need for more transparency. "We can assume that the larger social media companies actively cooperate with state authorities," Weiss told DW in an email. "But without public information we can only speculate about the exact extent and form of the cooperation. We know from the latest Google 'Transparency report' that the number of requests for information from state sources to social networks have been rising constantly in the past few years."
"Our public information about how often which data is called on, and on what basis, should not be based on voluntary reports from Internet firms," he added. "That is something that the respective authorities should be obliged to disclose to the public."
Secrecy the sticking point
Carl Miller, research director at the Centre for the Analysis of Social Media (CASM) at UK think tank Demos, was also unsurprised by the revelations about Max Kelly. After all, where else are government agencies going to recruit experts from? "All the data mining and big data analytics technologies which are now required to understand social media - the expertise to conduct those, the data science, doesn't sit within government," he told DW. "We actually thought there needs to be a structure in place to allow government to draw on where this expertise is most concentrated, which really is in the private sector."
And yet, as Snowden's revelations show, there is a big gap between what is happening and what we are being told. Social media sites repeatedly assert that they only share data with government agencies when they are legally compelled to. But in fact, social media sites are being more proactive than that - according to the New York Times, they are putting together teams of in-house experts to find ways of making our data more directly available to government agencies.
Jury service for surveillance decisions?
Miller argues that the problem is not necessarily the surveillance programs themselves, but the secrecy that surrounds them - the fact that no one had even heard of PRISM or Tempora until last month. "Social media and intelligence can both pay great dividends to public safety and security, but at the same time it needs to be done in a publicly-argued framework," he said. "And in many cases it does feel like it is being done without the clear knowledge and consent of society, and that is definitely a cause for concern."
Miller can also see why security forces are so keen on mining data from social media. "The growth of social media means that we have transported our social lives to these digital platforms, and that means all of our social lives," he said. "All the good things - working together, cooperating, debating, but also the bits of our social lives that are illegal or problematic. There needs to be a response to this from the agencies which keep us safe. Social media is increasingly a forum for crime and facilitating criminals."
His own solution to the security vs. privacy balance sounds rather radical: "I think what's required is a more profound shake-up of oversight and surveillance," he said. "Maybe we need something that is more directly democratic - more like jury service. Where say 50 individuals are picked at random, cleared, and in our names for six months make decisions about whether the steps are being taken are necessary and proportionate."