Trevor Paglen - Skynet

A History of the World in Spy Objects, Episode 15: Trevor Paglen - Skynet
‍
NARRATOR: To understand the unbelievable significance of the item I'm about to show you - if it can even be described as an item - first I need to tell you a story. It’s about a man named Ahmad Zaidan. Until 2015, Ahmad worked in Islamabad, the capital of Pakistan, as the bureau chief for the Al Jazeera News Network. Back in 2010, he became well known for a series of interviews with Osama bin Laden. It was during that time that he landed on the radar of the NSA and CIA, who came to suspect him of belonging to al-Qaeda himself. But what drove the American intelligence agencies’ suspicions? It is all about data - metadata, to be precise -  the timing of phone calls, geolocation pings - that kind of thing. We all leave a metadata signature. And according to the US government, Zaidan was suspicious enough to earn him a spot on a ‘kill list’ - which is exactly what it sounds like. But who gathers that evidence? Who makes that call? As it turns out, it’s not a person at all. It’s a machine. A machine called Skynet. This program has been carefully researched by artist and author Trevor Paglen.

TREVOR PAGLEN: What the signature means in that phrase is a metadata signature. What that means is that if you have a certain series of data points around you, as far as the NSA and CIA were concerned, you could be eligible for assassination because your electronic fingerprint fit a profile of a generic terror suspect that they posited.

NARRATOR: Zaidan had the data points to make him suspicious. He had the right metadata signature to qualify him as a potential terrorist, according to the CIA’s machine learning program. But why? How?

TREVOR PAGLEN: How did they create those signatures? This is where the Skynet program comes in. What it does is essentially process data from cell phones. It looks for the numbers that different cell phones dialed, the time duration, and who called who; but it also looks for location data from cell phones. It looks for other anomalous activities related to cell phones. They look at how often cell phones are turned on or off. They look for cell phones whose SIM cards have been swapped out and even try to determine whether handsets are being swapped from person to person. 

NARRATOR: Zaidan had a history of meeting Taliban chiefs in Pakistan and Afghanistan, including Osama bin Laden. He was moving around, making phone calls in certain locations with certain people, maybe switching between phones or SIM cards. 

TREVOR PAGLEN: And by looking at all these points of data - who’s calling who? What are the locations of the devices? What other devices are they associated with? What are some of the behaviors associated with the person controlling that device? They can add up to a metadata signature that would qualify somebody with that metadata signature of being designated a terrorist and being assassinated with a drone.

NARRATOR: In other words, a machine-learning program dictates that certain people are monitored, hunted down, and potentially killed without trial.

TREVOR PAGLEN: This is done in a world of computers and machines and cell phone towers and AI algorithms which really don’t look like much at all in terms of the aesthetics of it. But at the end of the line, the aesthetics are quite intense, which is a drone showing up that may or may not be visible, and a missile being fired, and people being killed. Skynet in particular is about cell phones. And when you have access to that cell phone infrastructure you can see all of the phones that are associating with different towers. You can see the IDs of those phones. You can look at the call histories of those phones as well. What you do is you collect all the information on all the phones, and this is echoing Keith Alexander’s famous quote, “Collect it all. All the time.” The idea is then to put all of those into huge artificial intelligence models that would try to determine the various patterns and ‘signatures’ that would distinguish somebody who is a terror suspect from somebody who is not.

NARRATOR: Keith Alexander, who Trevor just mentioned, was the director of the NSA during the Bush and Obama administrations. His motto Collect It All defined his approach to security. What started as a radical method for obstructing bomb attacks against US military in Iraq became a pervasive policy applied to literally everyone. His goal was to collect, store, and monitor every phone and Internet communication that takes place on Earth. Intercepting communications between US citizens was - and is - supposed to be illegal without a court order. Intercepting communications between foreigners, though, was, and is, perfectly fine. The mobile phone data that was being monitored by the Skynet program belongs to about 50m Pakistani citizens. The idea was that by collecting all this data on millions of ordinary people, it would be possible to find the terrorists among them. 

TREVOR PAGLEN: When it was made public, there was a huge amount of … really anger about it for many, many different reasons. First reason was the theory that somebody who was a terrorist suspect would have a different kind of metadata signature than someone who wasn’t. And it would turn out that a lot of journalists would have the same kinds of metadata signatures.

NARRATOR: Ahmad Zaidan was one such journalist. The CIA pegged him as a member of al-Qaeda and the Muslim Brotherhood, as well as an employee of Al Jazeera. Bizarrely, no human being at the NSA queried his classification. No one pointed out that al-Qaeda and the Muslim Brotherhood are two totally different and opposing organizations. No one mentioned that he had consistently publicized his assignments and published his interviews with Osama bin Laden - not exactly the typical behavior of a terrorist. 

TREVOR PAGLEN: In order to make machine-learning models work at scale you need massive, massive, massive amounts of data and one of the problems from the scientific perspective is that there are actually very few terrorists in the world, and there are even fewer metadata signatures of terrorists, so it is really not clear how you would have enough of that information to train one of these models. 

NARRATOR: And the thing is, baked into the algorithms of Skynet, is an accommodation for false positives. In other words, a machine-learning program dictates that certain people are monitored, hunted down, and potentially killed without trial. Zaidan found all this out when Edward Snowden leaked classified NSA documents to the press in 2013. As he later wrote: “The allegations against me put my life in clear and immediate danger when we consider that many people have lost their lives as a result of such fake information.” According to the Bureau of Investigative Journalism, by 2016, somewhere between 2,500 and 4,000 people had been killed by the US drone strikes in Pakistan. As many as 900 of those may have been civilians. But their metadata signatures signed their death warrants. Zaidan brought a case against the US government seeking a court order for his removal from the US ‘kill list’. The case was ultimately dismissed. The court claimed Zaidan was not at immediate threat as a result of being included on the Skynet list.

TREVOR PAGLEN: This Skynet program, as it was being conducted in the 2000s, was a military intelligence assassination program but many of those techniques are becoming more and more a part of our everyday lives and more and more a part of the infrastructure and institutions that we all interact with.

NARRATOR: These machine-learning programs may have been created with military surveillance in mind but their impact spreads far beyond strategic intelligence.

TREVOR PAGLEN: We are looking towards a future that is already very much present, in which everybody’s metadata signatures are going to have an effect on their credit rating, on how much they may pay for car insurance or health insurance, on their ability to get credit, the amount of attention they get from the police. Again, it was a machine-learning program that is very similar to the kinds of machine-learning systems we now find run by the Googles, Amazons, and the Facebooks of the world. We see the same kinds of problems of classification and the same kinds of problems around due process in particular.

NARRATOR: Machine-learning models do indeed now influence many areas of our lives, some for better, some for worse but hopefully they won’t lead to us being put on a Kill List. And yet it’s a reality that’s all too easy to ignore… unless you find yourself being placed on the wrong list. I’m Alice Loxton. More secrets await in the next episode of A History of the World in Spy Objects.

‍