Google fraud dashboard accuracy records
| Submitted | Jan. 18, 2024 |
| Est. Completion | March 1, 2023 |
MuckRock users can file, duplicate, track, and share public records requests like this one. Learn more.
Communications
From: Todd Feathers
To Whom It May Concern:
Pursuant to the Ohio Open Records Law, I hereby request the following records:
1) Records documenting the accuracy of JFS's unemployment fraud scoring algorithm (the "Google fraud algorithm") since January 1, 2021. This should include documents and/or data that record the algorithm's AUC, its false positive rate, its margin of error, indications of how strongly its fraud scores correlate to proven fraud, and/or other statistical measurements of accuracy.
Records previously released to me by JFS show that the department measures and has recorded the accuracy of the Google fraud algorithm. For example:
- The Technical Design Document for the Google fraud algorithm states that "high scores are strongly correlated with fraud." (Page 7)
- A June 5, 2021 presentation to the ODJFS Unemployment Insurance Public-Private Partnership states that claims assigned fraud scores between 300 and 550 have a "50/50 likelihood of fraud." (slide 26)
While it is possible that records responsive to this request will contain material that JFS believes is exempt from disclosure, please note that the department is still required to provide the non-exempt portions of the records. "If a public record contains information that is exempt from the duty to permit public inspection or to copy the public record, the public office or the person responsible for the public record shall make available all of the information within the public record that is not exempt" ORC Section 149.43(B)(1). And as noted above, the department has already released portions of public records containing details of the algorithm's accuracy.
I ask that all fees be waived as I am a journalist and intend to used the requested records to publish articles in the public interest and not for any commercial purpose.
If you choose to impose fees, I request a detailed breakdown of the fees, including the hourly wage of each employee involved and an explanation of the employee hours required to fulfill the request.
If you choose to reject this request or redact portions of responsive documents, I ask that you provide a detailed breakdown of the statutory exemptions and associated case law underlying your decision to withhold each/any portions from public review.
In the event that there are fees, I would be grateful if you would inform me of the total charges in advance of fulfilling my request. I would prefer the request filled electronically, by e-mail attachment if available or CD-ROM if not.
Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response to this request within 10 business days.
Sincerely,
Todd Feathers
From: Department of Jobs and Family Services
We are in receipt of your request.
[cid:image001.png@01DA49F6.4DBE23A0]
Bill Teets
Communications Director
30 East Broad Street, 32nd Floor
Columbus, Ohio 43215
C: 614.403.2331
-
image001
From: Department of Jobs and Family Services
Dear Mr. Feathers, We have received your public records request and will begin working on it.
[cid:image001.png@01DA4A2D.9BBA42F0]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/hers
-
image001
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following Ohio Open Records Law request, copied below, and originally submitted on Jan. 18, 2024. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Department of Jobs and Family Services
Mr. Feathers, We are working on your request and will likely have a response to you by early next week. I apologize for the delay.
[cid:image001.png@01DA5854.33F45A40]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/hers
-
image001
From: Todd Feathers
Thank you very much, Nicole!
Todd
From: Todd Feathers
Hi Nicole,
Checking in on this request to see if you had a new estimate for when it will be completed?
Thanks very much,
Todd
From: Department of Jobs and Family Services
Mr. Feathers, I am so sorry. I have been working on the request—it is large and requires substantial review—and then was unexpectedly away from the office for the last two days. I plan to work on it over the next few days and, again, am hopeful to have it to you by the end of next week.
[cid:image001.png@01DA60C4.9ED22680]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/hers
-
image001
From: Department of Jobs and Family Services
Mr. Feathers, I will be sending you by File Drop five reports from the Russell Allen group. They are out of sequential order, as I am trying to get through some of the lengthier documents as soon as possible. There are approximately 20 more documents that I need to review. I am hoping to have that completed by the first week in March, if not sooner.
You will note that redactions have been taken on these records. I redacted content when it either appeared to be case information from our systems that is protected by federal law, primarily 20 CFR 603.4, O.R.C. 4141.21 and 4141.211, or where the content provided details about how our systems function to identify and mitigate fraud. That type of content is excluded from release as a public record pursuant to O.R.C. 149.433(A) because it documents computer infrastructure. It may also be protected as proprietary software and as a potential security threat. O.R.C. 149.43 and 149.433 (A)(1); see also, State ex rel. Gambill v. Opperman, 135 Ohio St.3d 298, 2013-Ohio-761.
[cid:image001.png@01DA64E1.A8CB35C0]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/her
-
image001
From: Department of Jobs and Family Services
This is an automated message sent to you by the FileDrop service.
Moss, Nicole <Nicole.Moss@jfs.ohio.gov> has dropped off 6 files for you.
IF YOU TRUST THE SENDER, and are expecting to receive a file from them, you may choose to retrieve the drop-off by clicking the following link (or copying and pasting it into your web browser):
https://filedrop.cloudfs.ohio.gov/pickup?claimID=CXTe6254MR9oDhga&claimPasscode=KEcV2XcNJ6NNsx4p&emailAddr=88488
You have 7 days to retrieve the drop-off; after that the link above will expire. If you wish to contact the sender, just reply to this email.
The sender has left you a note:
Mr. Feathers, These are the records that I mentioned in my email you a few minutes ago. There are, however, six records, not five.
Nicole Moss
Full information about the drop-off:
Claim ID: CXTe6254MR9oDhga Claim Passcode: KEcV2XcNJ6NNsx4p Date of Drop-Off: 2024-02-21 16:25:39
-- Sender --
Name: Moss, Nicole Organization: State of Ohio Email Address: Nicole.Moss@jfs.ohio.gov -- Files --
Name: P3 Project Sprint #8_2021_1002_Redacted.pdf Size: 14.7 MB SHA-256 Checksum: 728A839B351E679EF7762C08EBC95D9F1B878FF28C3F43C2C8A54638CEAFFDB3 Content Type: application/pdf Name: P3 Project Sprint #9_2021_1019.pptx Size: 737.4 KB SHA-256 Checksum: 944CEC728AABDBC6ADCF8B51F70E6E2ECACCDAC2B4ECDAC6B4DB6142E68BD80F Content Type: application/vnd.openxmlformats-officedocument.presentationml.presentation Name: P3 Project Sprint #19_FINAL_Redacted.pdf Size: 11.1 MB SHA-256 Checksum: A34D142B46958A0DAE4441F57362B90A7F6CDC89983477469E3A300478DC9131 Content Type: application/pdf Name: P3 Project Sprint #20_Redacted.pdf Size: 7.5 MB SHA-256 Checksum: 3D3D44564CE486BB02AE3DA2D7AAB4ECB5B082E34B07E257614A197589218B78 Content Type: application/pdf Name: P3 Project Sprint #21_Redacted.pdf Size: 16.6 MB SHA-256 Checksum: 94DE5EC2CF3FFF8918210C0D1E72BF881A4A6C19AE6A1EBC970591426F53CDDA Content Type: application/pdf Name: Sprint #6_0805_Redacted.pdf Size: 7.2 MB SHA-256 Checksum: 1E67668BCDE89F5E5DB5E36ECE2D353CA881C02CEED8EB4B45186ED8BD88FD63 Content Type: application/pdf
From: Todd Feathers
Thank you very much Nicole, I appreciate the updates and rolling doucment production!
Best,
Todd
From: Department of Jobs and Family Services
Mr. Feathers, I will be sending you another five reports by File Drop. There are redactions on these as well, and the basis for the redactions is the same as below.
[cid:image001.png@01DA6BEF.F5CF72E0]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/her
-
image001
From: Department of Jobs and Family Services
This is an automated message sent to you by the FileDrop service.
Moss, Nicole <Nicole.Moss@jfs.ohio.gov> has dropped off 5 files for you.
IF YOU TRUST THE SENDER, and are expecting to receive a file from them, you may choose to retrieve the drop-off by clicking the following link (or copying and pasting it into your web browser):
https://filedrop.cloudfs.ohio.gov/pickup?claimID=stxuRTjRQVB3TdJw&claimPasscode=Yz4riS8dApTJCTRb&emailAddr=89575
You have 7 days to retrieve the drop-off; after that the link above will expire. If you wish to contact the sender, just reply to this email.
The sender has left you a note:
Mr. Feathers, These are the five reports that I mentioned to you.
Nicole Moss
From: Todd Feathers
Hi Nicole,
Thanks again for your continued help on these requests. Quick question for my record keeping:
This batch of records you sent today and the last batch (all containing sprint presentations from the UI public private partnership) are responding to my previous request (sent 8/15/23)for records from the public private partnership, correct? And they aren't in response to my latest request for records regarding the accuracy of the JFS' fraud detection system?
It's no issue on your end, but the communications sometimes get assigned to the wrong request on MuckRock.
Much appreciated!
Todd
From: Department of Jobs and Family Services
Mr. Feathers, I will be sending you by File Drop the final set of reports from the Russell Allen group.
You will note that redactions have been taken on these records. I redacted content when it either appeared to be case information from our systems that is protected by federal law, primarily 20 CFR 603.4, O.R.C. 4141.21 and 4141.211, or where the content provided details about how our systems function to identify and mitigate fraud. That type of content is excluded from release as a public record pursuant to O.R.C. 149.433(A) because it documents computer infrastructure. It may also be protected as proprietary software and as a potential security threat. O.R.C. 149.43 and 149.433 (A)(1); see also, State ex rel. Gambill v. Opperman, 135 Ohio St.3d 298, 2013-Ohio-761. Finally, redactions were taken pursuant to O.R.C. 149.43(A)(2) on records that pertain to a law enforcement matter and have a high probability of disclosing an uncharged suspect. State ex rel. Musial v. N. Olmsted, 106 Ohio St.3d 459, 2005-Ohio-5521.
We are also still working on the information about the algorithm that you have requested and hope to have something for you within the next 10 days.
[cid:image002.png@01DA1B91.F0F5F630]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/her
This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain private, confidential, and/or privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, employee, or agent responsible for delivering this message, please contact the sender by reply e-mail and destroy all copies of the original e-mail message.
-
Picture (Device Independent Bitmap) 1
From: Department of Jobs and Family Services
This is an automated message sent to you by the FileDrop service.
Moss, Nicole <Nicole.Moss@jfs.ohio.gov> has dropped off 13 files for you.
IF YOU TRUST THE SENDER, and are expecting to receive a file from them, you may choose to retrieve the drop-off by clicking the following link (or copying and pasting it into your web browser):
https://filedrop.cloudfs.ohio.gov/pickup?claimID=zvFBQv7mYxgYog5R&claimPasscode=C6ecceeP7vFW5W8E&emailAddr=90434
You have 7 days to retrieve the drop-off; after that the link above will expire. If you wish to contact the sender, just reply to this email.
Full information about the drop-off:
Claim ID: zvFBQv7mYxgYog5R Claim Passcode: C6ecceeP7vFW5W8E Date of Drop-Off: 2024-03-08 15:15:10
-- Sender --
Name: Moss, Nicole Organization: State of Ohio Email Address: Nicole.Moss@jfs.ohio.gov -- Files --
Name: P3 Project Sprint #15_Redacted.pdf Size: 6.4 MB SHA-256 Checksum: D5537292ED9CC7EA0A1256A99121A4414D0E146341247FA24D340F387430A507 Content Type: application/pdf Name: P3 Project Sprint #16_Redacted.pdf Size: 5.3 MB SHA-256 Checksum: 743CBB7B757DC360E7516515B61E8BF054920368DB0BF2F480155E30D61FCE49 Content Type: application/pdf Name: P3 Project Sprint #17.pptx Size: 1.1 MB SHA-256 Checksum: 13710F2F9D6C8277BCD32417733661142A62A1957559D9743394864EE8F23F9B Content Type: application/vnd.openxmlformats-officedocument.presentationml.presentation Name: P3 Project Sprint #18 (1).pptx Size: 1.3 MB SHA-256 Checksum: EA3C8A2D7ECA38729187D2687CBD67DE0339321C55293AABBF646BFD94E61E0F Content Type: application/vnd.openxmlformats-officedocument.presentationml.presentation Name: P3 Project Sprint #20_Redacted.pdf Size: 7.7 MB SHA-256 Checksum: 5C2CE493038BD157A2D26EF3EB11620F021D4BAEB68E2BA33122BB8AE175B1BA Content Type: application/pdf Name: P3 Project Sprint #21_Redacted.pdf Size: 16.5 MB SHA-256 Checksum: 05B1ECBA9433A1D0432E7761E8AE7187FC6E5B49995250825EC0C63A9F4CE4D1 Content Type: application/pdf Name: P3 Project Sprint #22_Redacted.pdf Size: 5.6 MB SHA-256 Checksum: 3D08FD01C8AF6ACDF21A3084978791C6996B627D6A3BC4F67EC2F4E0A14521DD Content Type: application/pdf Name: P3 Project Sprint #23_Redacted.pdf Size: 5.7 MB SHA-256 Checksum: 50D4486C5AC557608005B74520AF17E52A10E2593A76101D86FBA3E4E9F60993 Content Type: application/pdf Name: P3 Project Sprint #24_Redacted.pdf Size: 6.7 MB SHA-256 Checksum: 9DD976D08A28A030F80213900052A484099FE98DE1F3B4CF0DE8BC6ECB67C6FE Content Type: application/pdf Name: P3 Project Sprint #25.pptx Size: 7.5 MB SHA-256 Checksum: 35F14AE17DF81B03DF2BF2D458A134391556967EB5B52C0AFFA33599CCC44084 Content Type: application/vnd.openxmlformats-officedocument.presentationml.presentation Name: P3 Project Sprint #26_Redacted.pdf Size: 3.9 MB SHA-256 Checksum: 84BBA4AA22163FF8E0CE08A8BEAADA2A68B04D155F0B6576F8662D358FE0A499 Content Type: application/pdf Name: P3 Project Sprint #27 Funds Recover.pptx.pdf Size: 321.4 KB SHA-256 Checksum: 23310397AAE3326C4950FBB5EDE45C2336B519A545888B7A6B8875B9FF827001 Content Type: application/pdf Name: RAP Sprint #7_0902.pdf Size: 1.9 MB SHA-256 Checksum: 726C7FD0CFD0DC2F4813FA10B25AD9125D22CE0E2026FEDC6980D212EDC04750 Content Type: application/pdf
From: Department of Jobs and Family Services
Mr. Feathers, Can you please do me a favor? I need you to identify a document that you’ve asked about. Your request about algorithms is related to “The Technical Design Document for the Google fraud algorithm states that "high scores are strongly correlated with fraud." (Page 7).” I am wondering if you can either send this document back to me or give the date it was produced. I am unable locate it and it may be helpful in finding out the supporting documentation for the algorithm.
[cid:image001.png@01DA7171.699F6D70]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/Her
This e-mail, together with any attachments or files transmitted with it, may contain confidential information belonging to the sender, or constitute non-public information that is not subject to disclosure under O.R.C. Section 149.43, Ohio's Public Records Act. Additionally, if the sender is an employee of the Ohio Department of Job and Family Services, this e-mail, together with any attachments or files transmitted with it, may contain information that is protected by the attorney-client privilege or other applicable privileges. The information in this e-mail is intended solely for the use of the individual or entity to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you have received this e-mail in error, please immediately notify the sender by e-mail and delete this e-mail from your system. If you are not the intended recipient, do not disclose, copy, distribute, or re-distribute this e-mail or take any action in reliance on the contents of the information contained in this e-mail. Nothing contained in this notice by itself exempts, nor should it be interpreted to by itself exempt, this e-mail from disclosure as a public record upon the proper submission of a request for public records pursuant to O.R.C. Section 149.43, Ohio's Public Records Act.
P Please consider the environment before printing this email.
-
image001
From: Todd Feathers
Hi Nicole,
The Technical Design Document is attached. The relevant sentence is on page 7, at the end of the "Solution Design" section.
Best,
Todd
From: Muckrock Staff
To Whom It May Concern:
I'm following up on the following Ohio Open Records Law request, copied below, and originally submitted on Jan. 18, 2024. You had previously indicated that it would be completed on March 1, 2023. I wanted to check on the status of my request, and to see if there was a new estimated completion date.
Thanks for your help, and let me know if further clarification is needed.
From: Department of Jobs and Family Services
Mr. Feathers, We are still actively working on this part of your records request. I will try to update you on the status of the request by close of business tomorrow.
[cid:image001.png@01DA7EA5.FE033E60]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/her
-
image001
From: Todd Feathers
Gotcha, thank you, Nicole!
Todd
From: Department of Jobs and Family Services
Mr. Feathers, I should have more information for you about when we will be able to respond by the end of the week.
[cid:image001.png@01DA7F84.458CB1B0]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/her
-
image001
From: Department of Jobs and Family Services
Mr. Feathers, I am writing to say that I believe I will be able to finally respond to the remaining part of your public records request below later next week. You have been very patient and that is so appreciated.
[cid:image001.png@01DA8674.6E29C350]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/Her
This e-mail, together with any attachments or files transmitted with it, may contain confidential information belonging to the sender, or constitute non-public information that is not subject to disclosure under O.R.C. Section 149.43, Ohio's Public Records Act. Additionally, if the sender is an employee of the Ohio Department of Job and Family Services, this e-mail, together with any attachments or files transmitted with it, may contain information that is protected by the attorney-client privilege or other applicable privileges. The information in this e-mail is intended solely for the use of the individual or entity to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you have received this e-mail in error, please immediately notify the sender by e-mail and delete this e-mail from your system. If you are not the intended recipient, do not disclose, copy, distribute, or re-distribute this e-mail or take any action in reliance on the contents of the information contained in this e-mail. Nothing contained in this notice by itself exempts, nor should it be interpreted to by itself exempt, this e-mail from disclosure as a public record upon the proper submission of a request for public records pursuant to O.R.C. Section 149.43, Ohio's Public Records Act.
P Please consider the environment before printing this email.
-
image001
From: Todd Feathers
You've been doing a lot of work on my behalf, Nicole, so the appreciation is all mine!
Todd
From: Department of Jobs and Family Services
Good morning, Mr. Feathers, The Ohio Department of Job and Family Services has conducted a thorough search and has determined that it does not have records that are responsive to this request:
1) Records documenting the accuracy of JFS's unemployment fraud scoring algorithm (the "Google fraud algorithm") since January 1, 2021. This should include documents and/or data that record the algorithm's AUC, its false positive rate, its margin of error, indications of how strongly its fraud scores correlate to proven fraud, and/or other statistical measurements of accuracy.
[cid:image001.png@01DA8CB4.D9513230]
Nicole S. Moss
Deputy Chief Counsel, Office of Legal Services
D: 614.995.0131
She/Her
This e-mail, together with any attachments or files transmitted with it, may contain confidential information belonging to the sender, or constitute non-public information that is not subject to disclosure under O.R.C. Section 149.43, Ohio's Public Records Act. Additionally, if the sender is an employee of the Ohio Department of Job and Family Services, this e-mail, together with any attachments or files transmitted with it, may contain information that is protected by the attorney-client privilege or other applicable privileges. The information in this e-mail is intended solely for the use of the individual or entity to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you have received this e-mail in error, please immediately notify the sender by e-mail and delete this e-mail from your system. If you are not the intended recipient, do not disclose, copy, distribute, or re-distribute this e-mail or take any action in reliance on the contents of the information contained in this e-mail. Nothing contained in this notice by itself exempts, nor should it be interpreted to by itself exempt, this e-mail from disclosure as a public record upon the proper submission of a request for public records pursuant to O.R.C. Section 149.43, Ohio's Public Records Act.
P Please consider the environment before printing this email.
-
image001
Files
pages