When New York City announced a group in early 2018 to study and establish processes around artificial intelligence programs, the largest city in the country seemed ready to create a framework to follow for the hundreds of others quickly adopting automated decision systems (ADS).
But the final NYC ADS Task Force report, released last month, concluded that quick answers wouldn’t be available for addressing questions of fairness and accountability in ADS, even when trying to define the term itself. Comprised of academics, technologists, and city representatives, the ADS task force met multiple times over its 18-month existence, ultimately deciding ongoing, centralized oversight will be required to develop legal options or standards for those who wish to interrogate or challenge ADS use.
In its final report, the task force focused on three areas for action: centralization of city oversight of ADS, expanded public education on its use, and further formalization of ADS management. It made no specific recommendations for addressing bias or other concerns that spurred its creation, settling on a framework of questions for considering and prioritizing further scrutiny of uses of AI.
From street corners to court rooms, important government decisions are made each day by computer calculation, using data analytics to predict or identify certain needs or risks. Something as seemingly non-threatening as an Excel formula could have life-altering effects if used, for example, to weight different factors determining government benefit eligibility. Similarly, other algorithms or programs may assign to individuals scores rating their risk of recommitting an offense, discriminating against their tenants, or suffering from a particular disease.
Actual uses of ADS in NYC weren’t definite, even for group members, and some saw the city’s reluctance to provide a list of know automated systems as a hindrance to the whole process.
Meredith Whittaker, co-founder of the AI Now Institute at New York University, offered one of the most public critiques of the task force process, in particular its accessibility to the public and the implication that the broad ADS definition was an obstacle, writing:
“Non-governmental Task Force members, including myself, repeatedly argued that a broad definition of ADS is important, and that the context/purpose of use for a given systems[sic] is critical to guiding a determination of whether it constitutes an ADS or no. This would *include spreadsheets, and other multi-purpose technologies, among the technical systems that could qualify as an ADS.”
I'm a member of the NYC Automated Decision Systems Task Force. Yesterday the City released the Task Force report.
— Meredith Whittaker (@mer__edith) November 20, 2019
The report reflects the views of the City, not a Task Force consensus.
I sent comments to the City and asked that they be included. They weren't. Here they are: 1/ pic.twitter.com/QOFy00vxiv
In April, AI Now released its own list it had compiled of ADS in use by the City of New York. MuckRock has filed requests for any available records related to these ADS systems, which you can find with all of our coverage on the development of government AI via our project Algorithmic Control.
As the report notes, the use cases for these processes are still evolving, and setting regulations proved difficult in the growing library of potential scenarios for misapplication and abuse. The scope of potential AI processes complicated things early on for members of the committee, which, as the report noted, suffered challenges due to the “absence of a shared language on ADS.”
Ultimately, the group never reached a consensus on a definition, and hopes that specific recommendations might affect specific systems went unrealized, leaving it to other government members to push the conversation forward.
Shortly after its release, NYC Mayor Bill de Blasio issued an Executive Order establishing an Algorithms Management and Policy Officer, “to create a centralized resource on algorithms for agencies and the public alike, and continuing the important conversations that the Task Force initiated.” Last week, City Councilor Peter Koo introduced another local bill requiring each city department to generate an annual list of ADS in use and how.
“The Automated Decision Systems Task Force was the first of its kind in the country, and it tackled the complex issues around the systems and tools our city uses to help us make decisions that affect millions of New Yorkers,” Laura Feyer, a NYC spokesperson, told MuckRock. Feyer pointed out that specific examples in transportation, policing, education, and emergency management had been reviewed by the group. “While not everyone agreed on every issue, after 18 months the final report reflected an overwhelming consensus on key, actionable recommendations for the City.”
You can read the full report below:
Image via flickr licensed under CC BY 2.0
Algorithmic Control is part of a joint research and reporting project from MuckRock and the Rutgers Institute for Information Policy and Law. Support for this project is provided by Rutgers, but the reporting is editorially independent. View the full database of requests, learn more about the project, or get in touch with the reporter.
Algorithmic Control by MuckRock Foundation is licensed under a Creative Commons Attribution 4.0 International License.
Based on a work at https://www.muckrock.com/project/algorithmic-control-automated-decisionmaking-in-americas-cities-84/.
