DocumentCloud has had many feature updates within the last couple of months, including the ability to de-index documents from DocumentCloud’s public search and search engines like Google, the ability to upload documents via email, several new Add-Ons and pro features which include the ability to search publicly accessible notes and use Amazon’s Textract OCR to get better text extraction from within hard-to-OCR documents.

Uploading large sets (hundreds, thousands, or even millions) of documents to DocumentCloud using the user interface can be laborious and requires careful monitoring of uploads for processing errors and splitting up the document set into smaller batches.

DocumentCloud’s Batch Upload Script was initially written to upload the CIA Crest files, which contains almost 1 million files. It keeps track of which files were uploaded successfully, so that it can be stopped and restarted and it will pick up where it left off, and errors can be retried. It uploads files in batches. It can be stopped gracefully by pressing CTRL+C (once) while it is running. A recent rewrite allows the script to run on any directory of documents.

Ongoing support program protects endangered materials through decentralized storage while giving DocumentCloud users a range of new features.

The comments from a senior New York City environment official overseeing its wastewater surveillance program represent a sharp departure from a joint statement made to MuckRock and the Gothamist last month by the city’s health and environment agencies, which called wastewater surveillance a “developing field,” stressing a need for further research before it could be used to inform policy action.