Uploading large sets (hundreds, thousands, or even millions) of documents to DocumentCloud using the user interface can be laborious and requires careful monitoring of uploads for processing errors and splitting up the document set into smaller batches.

DocumentCloud’s Batch Upload Script was initially written to upload the CIA Crest files, which contains almost 1 million files. It keeps track of which files were uploaded successfully, so that it can be stopped and restarted and it will pick up where it left off, and errors can be retried. It uploads files in batches. It can be stopped gracefully by pressing CTRL+C (once) while it is running. A recent rewrite allows the script to run on any directory of documents.

In 1978, the director of the CIA warned that excessive, impulsive secrecy was a danger — not only to the public’s right to know, but to the agency’s ability to keep the important secrets. 40 years later, that lesson still needs repeating.

Heading to New Orleans for #ONA2019 and not sure what to wear in Louisiana in the fall? Well, you’re in luck. The Society of Former Special Agents of the Federal Bureau of Investigation had their annual conference in NOLA in 1972, and their official publication, “The Grapevine,” has you covered. Literally.

There’s a running joke on both sides of the transparency community that the standards for secrecy are so absurd that “you could easily classify a ham sandwich.” And nowhere does that dictum ring more true than in regards to the Central Intelligence Agency, which has, on multiple occasions, classified ham sandwiches.

During my time at MuckRock, I’ve written extensively about the triumphs and tragedies of the Central Intelligence Archive cafeteria(s), including such FOIA favorites as “The Jazz Salad Incident,” “Bacon Accounting,” and “That Scene From Animal House But It’s All The Guys Who Couldn’t Kill Castro.” Before I go, I wanted to share one of my favorite finds from the CIA archives: The description of an employee’s aborted attempt to smuggle chicken out of the cafeteria in her purse.

An undated regulation uncovered in the Central Intelligence Agency archives, formerly classified SECRET, appears to outline the “Dos and Don’ts” for Agency historians. While most of the consideration goes into avoiding exposing the identities of undercover agents (and acknowledging the inherent difficulties therein), one surprising paragraph instructs historical officers to avoid “embarrassing incidents” or “unflattering statements” unless absolutely necessary.

In July of 1955, Lewis Strauss, chairman of the Atomic Energy Commission, wrote to CIA director Allen Dulles over matters of mutual interest. In one of those letters, uncovered in the Agency’s archives, Strauss thanked Dulles for a package he had sent him, using deliberately vague terms to describe its contents as to “avoid classifying this letter.” Strauss’ efforts were in vain however. Not only was the letter classified for just shy of 50 years, but the vague descriptor itself remains classified to this day.

A MuckRock user found an interesting memorandum from the late ‘70s in our recent assignment From the Archives: Memos mentioning Senator Joe Biden, in which the Central Intelligence Agency bemoaned the “burdens” placed on it by the recently strengthened Freedom of Information Act.

Did “A Russian Journal” author John Steinbeck offer to pass on information to the Central Intelligence Agency? According to a 1952 letter in the Agency archives from former Director of Central Intelligence Walter Bedell Smith, yes.

Partially due to Joe Biden’s time on the Senate Select Committee for Intelligence, a search of the Central Intelligence Agency’s CREST database returns thousands of records - news articles, memos, letters, and more - referencing the Senator.