Uploading large sets (hundreds, thousands, or even millions) of documents to DocumentCloud using the user interface can be laborious and requires careful monitoring of uploads for processing errors and splitting up the document set into smaller batches.

DocumentCloud’s Batch Upload Script was initially written to upload the CIA Crest files, which contains almost 1 million files. It keeps track of which files were uploaded successfully, so that it can be stopped and restarted and it will pick up where it left off, and errors can be retried. It uploads files in batches. It can be stopped gracefully by pressing CTRL+C (once) while it is running. A recent rewrite allows the script to run on any directory of documents.

There’s a running joke on both sides of the transparency community that the standards for secrecy are so absurd that “you could easily classify a ham sandwich.” And nowhere does that dictum ring more true than in regards to the Central Intelligence Agency, which has, on multiple occasions, classified ham sandwiches.

During my time at MuckRock, I’ve written extensively about the triumphs and tragedies of the Central Intelligence Archive cafeteria(s), including such FOIA favorites as “The Jazz Salad Incident,” “Bacon Accounting,” and “That Scene From Animal House But It’s All The Guys Who Couldn’t Kill Castro.” Before I go, I wanted to share one of my favorite finds from the CIA archives: The description of an employee’s aborted attempt to smuggle chicken out of the cafeteria in her purse.

An undated regulation uncovered in the Central Intelligence Agency archives, formerly classified SECRET, appears to outline the “Dos and Don’ts” for Agency historians. While most of the consideration goes into avoiding exposing the identities of undercover agents (and acknowledging the inherent difficulties therein), one surprising paragraph instructs historical officers to avoid “embarrassing incidents” or “unflattering statements” unless absolutely necessary.

In July of 1955, Lewis Strauss, chairman of the Atomic Energy Commission, wrote to CIA director Allen Dulles over matters of mutual interest. In one of those letters, uncovered in the Agency’s archives, Strauss thanked Dulles for a package he had sent him, using deliberately vague terms to describe its contents as to “avoid classifying this letter.” Strauss’ efforts were in vain however. Not only was the letter classified for just shy of 50 years, but the vague descriptor itself remains classified to this day.