CIA and NSA first sought to exploit commercial databases in mid-80s

Open Source Intelligence (OSINT) is the least famous, least exciting, and most prevalent form of intelligence, covering any sources that are theoretically open to anyone, such as newspaper articles, published books, or social media posts. With the ubiquity of the internet, the use of such commercial databases is beyond routine for both the Intelligence Community and the government at large, but there was a time, however, where the mere interest was not only cutting edge, but problematic.

By June 1985, the NSA had been looking at the issue and decided to reach out to CIA about it. The NSA memo, originally marked For Official Use Only, was sent from the NSA Director directly to the CIA Director, proposed that they work with the Intelligence Handling Committee to find a way to exploit open-source, commercial databases. Unsurprisingly, NSA volunteered “to take the lead” in exploring the issue. To the NSA, the benefits were obvious. “The savings in time and manpower … have long been recognized.” They could give “up-to-the-minute information at the fingertips of analysts, technical personnel, and executives.”

Given this, it was only natural that the NSA would want “to use this open-source information by integrating commercial database-derived information into a variety of internal support programs.”

The NSA saw several obstacles, including costs and security. Significantly, in addition to accessing the information, they would have to devise ways to integrate it, and allow users to assimilate the information before they would be able to successfully exploit it. These challenges wouldn’t be unique to any of the agencies in the Intelligence Community, or even the rest of the government, which led the NSA to conclude that no one agency should have to deal with these issues alone.

Security would be the major issue, as it included not only the still-emerging issues of cybersecurity, but also deniability. How could the NSA and other agencies access these databases without giving away their interests? NSA’s Information Resources Management Organization had begun looking into “alternatives for rapid secure access to commercial databases, including data downloading, transmission through a secure gateway computer or physical data transfers.”

For its part, CIA agreed and was already working on some integrating some commercial databases into its work. The Foreign Broadcast Information Service (FBIS) was charged with collecting information from foreign public media, which the Agency felt included accessing foreign commercial databases. According to a formerly SECRET memo prepared for the CIA Director, the project had first begun in 1984.

The pilot project was fairly ambitious, especially for the time. Not only did it allow the Agency to collect information more quickly, the pilot project proved that the Agency could collect “new bibliographic and documentary information [that was] not normally available through regular means of acquisition.” By sampling “some 50 online databases contained in the French Telesystemes-QUESTEL network”, the project had found scientific, technical and economic information that met FBIS’ requirements.