Allan Lasser - Development

Since we work so closely, it’s easy for me to understand where the rest of the staff hits friction, then to design and develop solutions to reduce that friction. Michael, JPat, or Beryl will point out some tedious, manual pattern they follow to resolve a task, then I’ll automate the steps for them.

Usually, this reduces their interaction to setting some inputs and pushing a button. Other times, it just means collecting a bunch of disparate information and bundling it all up in one place.

Previously, all of the work regarding a request was dumped into the same email inbox. Now, activity on the site that we need to deal with - whether a failed fax, an agency response, a user asking for help, or anything else - creates a task that we can deal with on the site itself.

We wanted to develop the task system organically, and that’s exactly what we’ve done. The system started simple, but has grown to be very powerful. Establishing the framework was the hardest part - now we can add new tasks and improve new ones relatively easily.

I’m proud of what we’ve built so far, and I’m excited to see how it continues to grow.

LASSER OUT

Beryl Lipton - Tasks

Grinding behind the front pages of MuckRock requests runs a series of tubes that direct communications to and from each inquiry and the agency it deserves. Every response, appeal, fee, and outgoing mailing goes through MuckRock’s Task System.

Because each agency, request, and appeal has special quirks of their own, the multiple inboxes of MuckRock’s early days were a slog. But Allan’s efforts have streamlined the process, helping us to handle - from one place - the various actions we might need to take on a request.

Each request is assigned an individual email address, which makes keeping requests straight a lot easier. Unfortunately, agencies sometimes misdirect communications, landing them in our Orphan folder. Or something else goes wrong; from the Task System, we also tackle particular user issues.

On each request page, you’ll see a bright red “Get Help” button.

When it’s clicked, a window is generated for your message.

And when it’s sent, we receive it as a message in the Task System.

Not all agencies handle all communications electronically in 2016. In fact, some handle all communications in good old paper-and-postage-stamp fashion. For these folks, we print out their letters, stuff them in envelopes, and stamp them appropriately. (Federal appeals require both the envelope and the letter to be marked.)

The Task machine - simple as it may appear, it mightily slays thousands of FOIA requests each month.

JPat Brown - Scans

Despite the fact that this is 2016 - a mere three years from the dystopian future of Blade Runner - roughly a third of all the government communications MuckRock handles are sent via snail mail, and have to be hand-scanned and processed into the system. While a fair chunk of that comes from smaller municipalities that at least have some semblance of an excuse, some of the largest federal agencies - the Justice Department, for example - receive FOIA requests via email, but mail all their responses.

This can get super frustrating, especially when a certain Agency That Shall Not Be Named (rhymes with “Smate Smesmartment”) is yelling at you to provide them with a tracking number that’s currently in the back of a mail truck on I-95. But I digress.

Mail runs the gamut from tiny envelopes containing acknowledgement letters to boxes with hundreds of pages of responsive documents. Scanning is a two part process - first, you “scan” a document with your eyes and brain to identifying critical details, such as which request it corresponds to, what the tracking number, did they call Beryl “Mr. Lipton” again, etc.

On a good day, the agency will have made our job easy for us and included the correct MR number, which is how we internally sort requests. On a regular day, they forgot to include it, and we have to track it down. On the worst days, they assigned it the wrong number, and we have to track it down while in a bad mood.

After that, we assign it one of seven core statuses - Awaiting Response, Awaiting Appeal, Fix Required, Payment Required, No Responsive Documents, Rejected, and everybody’s favorite, Completed - type the corresponding acronym codes into fScanX and then scan in the traditional sense that involves a scanner. These are all then dumped on a server and then uploaded to the backend via sufficiently advanced technology that is indistinguishable from magic.

For the record, the largest notch on my (incredibly depressing) scanning belt was just shy of a thousand pages, which didn’t take me longer than ten minutes now that we have a scanner that doesn’t have to be fed one page at a time. To be honest, huge physical document dumps really aren’t that bad, and are getting increasingly rare as even the most luddite-esque agencies have started burning docs to a CD - it’s the envelopes that’ll kill ya. I’ll take 600 pages over 60 envelopes any day of the week.

Oh, and one last gripe - the State Dep - er, “Smate Smemsmartment,” staples all their cables together, which means having to remove them all manually with one of those bitey things. Unnamed Agency, Please, stop doing that.

Michael Morisy - Scaling

When we started MuckRock in 2010, our goals were pretty modest. We just wanted to test this idea that making it easier for people to file public records requests and share the results could have a positive impact. We didn’t know if anyone would use it, we didn’t know if agencies would respond, and we didn’t know if anyone besides us would find a database of requests and their results useful. We knew it was something we wanted, that we thought could be incredibly powerful for our own journalism, but that was about it.

So the site was relatively simple. The first iteration was just a box on a website that asked people to put the information they were interested in in a form, and that sent me an email, and then I manually turned that into a request. A lot of our early requests — we just supported Massachusetts and the Federal FOIA at first - I actually had to hand deliver and pick up in person. But people liked it!

How am I just now hearing about @MuckRockNews?! Very exciting for the FOIA nerd types: http://www.muckrock.com/blog/ w/ @documentcloud!— Brian Boyer (@brianboyer) September 16, 2010

That worked OK for a few dozen requests, but as we scaled up to hundreds and then thousands and then tens of thousands of requests, the challenges have been fascinating. Maybe 80% of the work we do is intentionally invisible to the end user.

@awinston wait, so has MuckRock been printing and mailing my requests every time? Damn.— Ingrid Burrington (@lifewinning) May 13, 2014

Yes, that includes printing and mailing dozens of requests each week, mostly to the New York Police Department. But it also includes new challenges, like using machine learning to automate classification of some responses (about 10% - 20% on a given day).

But more interesting than those challenges is how the community continues to come up with amazing ways to use the site, shining light on how government operates and making democracy a little more informed.

Awesome FOIA tool @MuckRock + Seattle activist @pmocek = Quick, smart FOIA requests for gov surveillance docs. https://t.co/xj9tnXuRt0— Christopher Soghoian (@csoghoian) October 8, 2014

Enabling that work, and watching it pay off again and again, is what keeps me motivated after six years and 20,000 requests.

Interns - General Sanity

Of course, none of this would be possible without the help of our phenomenal FOIA interns - like Lukas here - who keep us from being buried in a bureaucratic hellscape. To them, and our 3,000+ users who literally are the reason we do what we do, we want to say thanks, and we’re ready for 20,000 more.