!! History Commons Alert, Exciting News
Trailblazer was a participant or observer in the following events:
NSA servers used to collect and sift data. [Source: FrancesFarmersRevenge.com]The National Security Agency (see 1952) begins building a massive data-mining system, code-named “Trailblazer,” that is intended to sift through reams of digital communications intercepts and find nuggets of information relevant to national security. The program’s task is huge—to sort through the 2 million bits of data the NSA collects every hour—and one made even more complex by the relatively new types of wireless, Internet, cell phone, and instant messaging communications now becoming ever more commonplace. Trailblazer is strongly embraced by General Michael Hayden, who became the NSA’s director in March 1999. Hayden recognizes from the outset that the NSA is years behind the technological curve, and casts Trailblazer as the future of the agency’s intelligence gathering and sorting. In November 1999, Hayden makes Trailblazer the centerpiece of his “100 Days of Change,” his plan to transform the agency into a leaner, more efficient organization, fast-tracking the program to vault it ahead of other initiatives. “It was going to structure us to handle the digital revolution,” a former intelligence official will recall. But from the outset the program has problems: a meeting between NSA and other government officials in December 1999 is unpromising, and, according to one government oversight official, the program “kicked off with not a real great definition of what it was trying to achieve.” Program managers fail to define standard data formats to allow for the proper sorting of information. After six years, $1.2 billion in expenditures, and endless man-hours of work, the utterly failed program will be recognized as the “biggest boondoggle… in the intelligence community” (see January 2006). [Baltimore Sun, 1/29/2006]
Concerned that NSA post-9/11 surveillance operations violated the US Constitution, a senior NSA official reports on the program to House Intelligence Committee staff (see Before October 31, 2001), then retires. William Binney, a crypto-mathematician, had served in the NSA for 36 years. In 1997 he was made technical director of the World Geopolitical and Military Analysis Reporting Group, a 6000-employee unit that focused on signals intelligence (SIGINT) reporting and analysis. In the last part of his NSA career, Binney focused on dealing with the NSA’s problem of information overload, co-founding the Signals Intelligence Automation Research Center (SARC) and leading a 20-member team to develop a data-mining and analysis program called ThinThread. This program made it possible to “correlate data from financial transactions, travel records, Web searches, GPS equipment, and any other ‘attributes’ that an analyst might find useful,” and “could chart relationships among people in real time.” Unlike the NSA’s existing centralized data processing systems, ThinThread was able to identify useful or useless data as it was collected, reducing the overload problem. However, though it targeted foreign communications, ThinThread also intercepted those of Americans, and “continued documenting signals when a trail crossed into the US.” Binney incorporated measures to protect privacy, but NSA lawyers still considered the program too invasive, according to a 2011 article by Jane Mayer based on interviews with Binney and another NSA whistleblower, Thomas Drake. In 1999, NSA Director General Michael Hayden decided to fund a rival program, Trailblazer, which would be developed by defense contractors (see Late 1999). Trailblazer will be abandoned in 2006 as unworkable, after costing $1.2 billion (see January 2006). [New Yorker, 5/23/2011; Wired News, 2/15/2012; Democracy Now!, 4/20/2012] In 2002, three NSA whistleblowers—Edward Loomis, J. Kirk Wiebe, and Binney—will ask the Pentagon to investigate the NSA for wasting “millions and millions of dollars” on Trailblazer. [Nation, 3/26/2013]
Post-9/11 NSA Surveillance Expansion - Binney will tell Mayer that, after the 9/11 attacks, his people began coming to him, saying things like: “They’re getting billing records on US citizens! They’re putting pen registers [call logs] on everyone in the country!” James Bamford will interview Binney in 2012 and write, “At the outset the program recorded 320 million calls a day, [Binney] says, which represented about 73 to 80 percent of the total volume of the agency’s worldwide intercepts.” Binney has not been personally “read in” to this domestic surveillance program, but some members of his SARC team have, as their knowledge of ThinThread code was needed to set it up. Binney became convinced elements of ThinThread were being used, but without privacy protections, meaning US persons could be targeted. Soon after learning these things, Binney takes his concerns to the House Intelligence Committee (see Before October 31, 2001), and retires on October 31. He will tell Mayer, “I couldn’t be an accessory to subverting the Constitution.” Other sources support Binney’s account of this NSA data-mining and monitoring program (see After September 11, 2001, October 11, 2001, After September 11, 2001, Late September, 2001, and October 2001). However, the claim that NSA domestic surveillance was initiated only after, and in response to, 9/11 is contradicted by information indicating that domestic monitoring programs and activities were established and conducted prior to 9/11 (see Late 1999, February 27, 2000, December 2000, February 2001, February 2001, Spring 2001, and July 2001). [New Yorker, 5/23/2011; Wired News, 2/15/2012; Democracy Now!, 4/20/2012]
ThinThread 'Would Likely Have Prevented 9/11' - Despite ThinThread’s capacity to collect actionable intelligence, Hayden vetoed the idea of deploying the system three weeks before 9/11, in August 2001. According to the Loomis, Wiebe, and Binney, this decision “left the NSA without a system to analyze the trillions of bits of foreign SIGINT flowing over the Internet at warp speed, as ThinThread could do.” During the summer of 2001, when “the system was blinking red,” according to CIA Director George Tenet, the NSA “failed to detect critical phone and e-mail communications that could have tipped US intelligence to al-Qaeda’s plans to attack.” [Nation, 3/26/2013]
Entity Tags: Edward Loomis, World Geopolitical and Military Analysis Reporting Group, J. Kirk Wiebe, William Binney, Thomas Drake, House Intelligence Committee, James Bamford, Trailblazer, Jane Mayer, National Security Agency, Signals Intelligence Automation Research Center, Michael Hayden, Thinthread
Timeline Tags: Civil Liberties
The National Security Agency’s ‘Trailblazer’ program (see Late 1999), envisioned in 1999 as an overarching state-of-the-art data-mining system capable of sorting through millions of telephone and Internet communications and pluck out items relevant to national security and counterterrorism, is an abject failure, according to multiple sources and reports. The program has soaked up six years of effort and $1.2 billion in taxpayer dollars, with nothing to show except some schematic drawings and a few isolated technological and analytical gadgets, and little hope of much future progress. Matthew Aid, who has advised three federal commissions and panels investigating the 9/11 attacks, says that Trailblazer is “the biggest boondoggle going on now in the intelligence community.” Part of the problem is that over its six years of development, Trailblazer has passed through three separate NSA divisions, each with its own priorities and design goals. Its overseers have failed to exert the proper authority to clearly define the program’s goals and keep the project on track. In 2003, the NSA’s inspector general found that the program suffered from “inadequate management and oversight” of private contractors and overpayment for the work that was done. The lead private contractor for the project, Science Applications International Corporation (SAIC), has not provided the technical and managerial expertise necessary to create the system. While the Bush administration has touted the NSA wiretapping program (see December 15, 2005) as vital to protecting the nation from terrorism, it allows the agency to mismanage Trailblazer, in essence allowing the agency to go increasingly “deaf” as millions of items of unimportant information overwhelm the agency’s ability to sort out key bits of information, according to a government official. A Congressional investigation of intelligence failures surrounding the 9/11 attacks found that the NSA did not sift out “potentially vital” information that could have predicted or even prevented the attacks—a lapse that Trailblazer was intended to correct. Aid says that the problem is akin to searching for a needle in a haystack that doubles in size every few months. Intelligence experts say that the problem with Trailblazer is like deciding whether to keep a piece of mail or throw it out based only on what is on the outside of the envelope. Approximately 95% of the information gathered by the NSA is discarded without ever being translated from its original binary form; the remaining 5% is turned into plain text for human analysts to survey. Trailblazer was designed to sort through this information to identify patterns, keywords, and links to other data. The program would, in theory, translate all of the information into plain text or voice data, analyze the results to identify items of interest, store the results in an easily searchable database, and forward selected items to the appropriate analysts for follow-up. But after six years of work, there will still be no consensus among agency managers and experts as how to create a system to do this. Interestingly, another, less grandiose program, code-named Thinthread, appeared promising—a 2004 Pentagon report found that Thinthread could work better and be put to use more quickly than Trailblazer—but NSA managers disagreed with the Pentagon report and canceled Thinthread. Instead, Hayden pushed the agency to get Trailblazer up and running after the 9/11 attacks, cutting into time needed for review and corrections. Internal and external warnings that the program was going off-course were ignored; because of its secrecy and technological sophistication, neither Congress nor the NSA was able to effectively monitor the progress of the program’s development. And the agency lost track of much of the $1.2 billion that was allocated by Congress for the program. NSA Inspector General Joel Brenner blames the waste and inefficiency on “inadequate management and oversight.” As of 2006, the Government Accountability Office, the investigative arm of Congress, has not investigated Trailblazer simply because no one in Congress had asked it to. Because of the impact of the 9/11 attacks, and the war in Iraq, Congress has never seriously considered cutting back or reviewing any programs such as Trailblazer that might provide information on further terrorist attacks. [Baltimore Sun, 1/29/2006]
Receive weekly email updates summarizing what contributors have added to the History Commons database
Developing and maintaining this site is very labor intensive. If you find it useful, please give us a hand and donate what you can.
If you would like to help us with this effort, please contact us. We need help with programming (Java, JDO, mysql, and xml), design, networking, and publicity. If you want to contribute information to this site, click the register link at the top of the page, and start contributing.