How to Read a Bias Audit Summary

What a Published Bias Audit Summary Is Supposed to Tell You

Your vendor renewal is due in six weeks, and procurement just forwarded a link: "Bias Audit Summary — AEDT XYZ, Published March 2025." Your job is to decide whether this document tells you what you need to know before you sign another year of use on NYC-resident candidates. You open the PDF. There's a table. Some percentages. A date. No explanation of what any of it means for your hiring pipeline.

This is the moment most HR Directors either skim past the numbers and hope, or spend an hour with a calculator trying to figure out if 74% is a red flag. Neither is a comfortable place to run a compliance program from.

A published bias audit summary is a public record — it exists because Local Law 144 requires it, and it's meant to be legible to the employer relying on it and to the candidates it affects. But "meant to be legible" and "is legible" are different claims, and the gap between them is where most reviews go wrong.

By the end of this piece, you'll know which sections of a bias audit summary carry the compliance weight, how to run the numbers yourself in minutes, and which omissions should stop you before you renew.

This is an operational and documentation guide, not legal advice. If a summary raises a question about your specific obligations, confirm the answer with DCWP or your own counsel — nothing here substitutes for that.

What Local Law 144 Actually Requires the Summary to Report

Local Law 144 sets three core obligations on an employer using an AEDT — an automated employment decision tool, defined broadly as any computational process derived from machine learning, statistical modeling, data analytics, or AI that issues a score, classification, or recommendation used to substantially assist or replace discretionary hiring or promotion decisions. The employer must arrange an annual independent bias audit, post a public summary of the most recent audit results — including the date the AEDT was distributed for use — on its website, and give candidates or employees notice at least ten business days before the tool is used, with a path to request an alternative process or accommodation.

The summary itself is the artifact you're reading. It's supposed to carry the audit's selection rates and impact ratios by category, and the distribution date that tells you which version of the tool the numbers actually describe. Everything else — narrative framing, methodology paragraphs, vendor branding — is context around that core.

How to Read a Bias Audit Summary: Start With the Selection Rates

The first real content in most summaries is a table of selection rates: the percentage of candidates in each demographic category who were selected, scored favorably, or advanced by the tool. This is the raw material everything else is built from, so it's where a careful read starts.

Look for three things. First, which categories are broken out — sex categories, race/ethnicity categories, and, if the audit is done properly, the intersection of the two. Second, the sample size behind each rate — a selection rate calculated on a handful of applicants in a given category tells you far less than one calculated on hundreds. Third, whether the rates are scored against the same job or job group you're actually using the tool for. A summary covering a different role family than the one you deployed doesn't answer your question, even if it looks complete.

If the summary skips straight to a pass/fail statement without showing the underlying selection rates, that's a gap worth noting, not a reason to stop reading — go to the next section and see if the ratios are shown instead.

Checking the Impact Ratios Against the Four-Fifths Rule

Once you have selection rates, the summary should show — or let you calculate — the impact ratio: the selection rate for a given category divided by the selection rate for the highest-selected category in the same comparison. This is where the four-fifths rule comes in. Under the EEOC's Uniform Guidelines, a selection rate for any group that falls below 80% of the rate for the highest-selected group can indicate adverse impact.

Here's a worked example, using round numbers to show the method rather than a real audit's figures. Say the highest-selected category in a summary has a 50% selection rate, and another category has a 38% rate. Divide 38 by 50: the impact ratio is 76%. That's below the 80% threshold, which means this pairing would flag under the four-fifths rule and deserves a closer look — not an automatic conclusion of discrimination, but a documented reason to ask the vendor or your auditor a follow-up question.

A selection rate below 80% of the top-selected group's rate is the signal the four-fifths rule is built to catch — it's a screening threshold, not a verdict.

Running this math yourself, category by category, is the single most useful thing you can do with a published summary. If you want the mechanics broken down further, the four-fifths rule explainer and the impact ratio deep dive both walk through additional edge cases, including what to do when a category has too few applicants to calculate a stable rate.

Intersectional Categories: Where Summaries Often Go Thin

Local Law 144 calls for impact ratios by sex categories, race/ethnicity categories, and the intersection of the two — for example, selection rates for women who are Black or African American as a category distinct from "women" and "Black or African American" analyzed separately. This intersectional layer is where a lot of published summaries thin out. A summary might show clean sex-only and race/ethnicity-only tables and then either compress the intersectional rows or omit them where sample sizes get small.

This isn't necessarily a sign of bad faith — small intersectional cells produce statistically noisy rates, and some vendors choose to note that limitation rather than publish an unreliable number. But it is a gap you should be able to name when you see it, rather than assume the absence means everything checked out fine. Academic review of published LL144 audits has found that some summaries may under-report disparities specifically because of missing demographic data, opaque aggregation choices, and metrics that don't reflect how the tool is actually deployed day to day, according to the "Auditing the Audits" analysis published through ACM FAccT in 2025. Reading a summary well means asking whether the intersectional rows are there, and if they're not, asking why.

For a fuller treatment of how intersectional reporting is supposed to work and where it commonly breaks down, see the intersectional bias audit guide.

The Date Problem: Is This Summary Even Current?

Every summary should carry the date the AEDT was distributed for use — that's a distinct requirement from the audit date itself. Check both. An audit conducted in 2023 attached to a tool distributed in 2025 tells you the vendor may have updated the model since the last independent look at it, which matters more than the summary's polish or format.

Posting compliance generally has been uneven. A "Null Compliance" study of 391 employers found that only 18 had posted an audit report and only 13 had posted a required transparency notice, according to research published through ACM FAccT in 2024. If a vendor or employer's public page is missing a current summary altogether, that absence is itself the finding — you don't need a table to read that.

The mechanics of where and how a summary is supposed to be posted, and what counts as satisfying that requirement, are covered in the summary-of-results publishing guide.

Common Gaps and Red Flags to Watch For

A few patterns are worth flagging every time you see them: selection-rate tables with no visible sample sizes, impact ratios presented without the underlying rates, an audit date that predates the current tool version, missing intersectional rows with no explanation, and language that describes the audit's scope more narrowly than the way you're actually using the tool.

None of these automatically means the audit is invalid — audit quality and enforcement rigor vary, and independent review of the city's own oversight found real gaps in verification. New York State Comptroller auditors found more compliance issues among a sample of companies than the city agency's own review of those same companies did, and a majority of test calls meant to route AEDT-related complaints were found to be improperly handled rather than reaching the enforcing agency, per DLA Piper's 2026 analysis of that audit. The practical takeaway isn't that summaries are worthless — it's that nobody is checking your reading of it for you. That's your job, and it's worth doing carefully.

If you want a structured way to check a vendor's audit status against what they've actually published, the vendor bias audit verification guide walks through that process end to end.

Turning a Careful Read Into a Documented Rationale

Reading a summary well is the input. What you do with that reading — writing down what you checked, what you flagged, and what you asked the vendor before renewal — is the output that actually protects you. Civil penalties for LL144 violations run up to $500 for a first violation and $500 to $1,500 for each subsequent one, accruing per violation per day, according to the Office of the NY State Comptroller. That's not a reason to panic over one thin summary; it's a reason to have a record showing you looked.

The fastest way to build that record consistently, across every vendor and every renewal cycle, is to run the four-fifths math yourself rather than trust a pass/fail line in a summary. Our Four-Fifths Rule & Impact-Ratio Reading Guide gives you the worksheet and a companion calculator to check any summary's selection rates against the 80% threshold in minutes, with a place to log what you found and why you accepted or flagged it. Download it before your next renewal comes across your desk — it's the first concrete step toward a documented rationale you can actually defend.