We’re excited to share the new and improved DocumentCloud experience with you. After seven months of development, we’ve rebuilt the web application to be faster and more useful for your day-to-day tasks. We’ve also established a solid technical foundation to help us add and improve features faster than before, for years to come.

You may notice some of our changes immediately—like our improved UI—and feel the effect of others with time, as we release new features. If you’re a long-time DocumentCloud user, we think you’ll love using the new platform. If you’re a front-end or open-source developer, there’s never been a better time to start contributing to the project.

Our public beta is ready for you to use today at next.documentcloud.org. We’ll be releasing the experience for everyone in mid-November. Keep reading for an in-depth explanation for why we started this project, how we approached it, and how it succeeded.

Goals & Motivations

We began this project with clear goals:

• Clean up the user interface to provide a consistent experience that scales across devices.
• Render pages on the server, making them faster to load and easier to share.
• Simplify the application’s logic to make it easier to understand its structure, reactivity and state, and make it easier to support new features.
• Modernize our technologies and development practices to speed up development, ensuring high-quality, bug-free code from the start.

DocumentCloud has always been a single-page application. In fact, it was one of the first. Backbone.js, an early framework for highly-interactive client-side applications, was extracted from the original DocumentCloud codebase.

But that architecture came with tradeoffs. The entire app needed to download and run before users could even see anything – often taking many seconds or longer. JavaScript errors could leave users staring at a blank web page.

There are more subtle issues, too: A client-only application needs to load all its JavaScript and render components in an empty state, then request whatever data it needs and then render everything again as data comes in. This leads to a slower site and flashes of half-rendered pages. On top of that, it’s a headache for developers trying to determine what to show in each state, and to manage data coming in asynchronously.

SvelteKit elegantly server-renders individual pages, only sending the code and data necessary to render each page. The result is much lighter pages with much shorter load times. It also provides best practices and patterns for data loading, state management, and user interaction, leading us to write much simpler code that’s easier to reason about.

SvelteKit’s nested layout system lets us share and compose interface elements, helping us create an application that doesn’t “flicker” between pages and feels much more coherent than previous versions. Finally, it has a rich ecosystem of tooling that helps us build, test, and debug the application during development.

Principles & Practices

Svelte and SvelteKit are built around the core concept of components: reusable, self-contained blocks of HTML, CSS, and JavaScript. A SvelteKit application nests components all the way down. Early on, we established some core design principles to guide our development practice:

1. Components should be as composable as possible. A component should do one thing well, and complex functionality should be decomposed into multiple components.
2. Components should be as stateless and deterministic as possible. Provided data, a component should render predictably and without side-effects.
3. Components should be well tested and well documented.

Our project began by mocking up entire browser screens in Figma, which helped us to rapidly explore possibilities, understand the shape we wanted the final product to take, and excite the rest of the MuckRock team with our vision. We took those mockups and broke them down into the individual components we’d need to compose them.

With Storybook, we built each individual component in isolation. As we worked, we naturally created artifacts documenting and testing each of our components as we built them. This means by the time we render our components in our SvelteKit application, they look and work just as we expect.

Components load data and take action by sending requests to the powerful DocumentCloud API, which lives outside this project. Writing all our API requests as separate, well-tested functions, we kept our components isolated and stateless. We integrated TypeScript into our application since we knew all data coming from the API would be delivered in an expected format. TypeScript helps us further document our code by typing inputs and outputs, while helping us catch bugs caused by the most common programming errors, like typos.

All of our work was incremental through GitHub. It’s been possible to follow our entire development journey on the sveltekit branch of our public repository, muckrock/documentcloud-frontend. In each pull request (PR), we automatically run tests, type checks, and builds against the code coming in. This continuous integration process is supported by Chromatic, which checks and flags our Storybook files for changes, and Netlify, which creates a fully-functional preview deployment for each PR. These automatic processes support code-review, where each of us reads over the other’s code while asking questions, providing feedback, and suggesting changes.

We’ve set ourselves up for success. Taken together, these tools and practices have allowed us to develop with speed and agility. We’re able to quickly define what we need to build, rapidly build it, have certainty that it works as expected, and receive feedback before merging the change in. Guided by our principles, we’ve been able to qualitatively evaluate code quality and whether a design approach “works” in the best way possible.

A foundation for the future

Our goal with this migration isn’t just to give you a better looking, faster DocumentCloud. We want to make it easier to continue adding features, for open source developers to contribute to the codebase and for people to use the components we’ve built in ways we haven’t yet imagined.

By building on DocumentCloud’s API and using components that can be isolated and reused, we hope to make more of the code that powers DocumentCloud available for reuse outside of documentcloud.org.

Again, the new version lives at next.documentcloud.org. Please try it today, and leave us feedback using the button in the upper right corner.

DocumentCloud has passed through many hands since it launched in 2010: Jeremy Ashkenas, Amanda Hickman, Sam Clay, Ted Han, Nathan Stitt, Anthony DeBarros, Mitch Kotler and Dylan Freeman have all built and maintained versions of this critical piece of journalism infrastructure. (And Mitch and Amanda are still here.) A host of other contributors have developed a thriving ecosystem around it.

This is the third major rebuild of the site, and we hope this lays the foundation for its next decade of service.