Choosing an Independent Bias Auditor for Local Law 144

Your AEDT Vendor's Audit Report Just Landed — And You're Not Sure Who Wrote It

Your applicant-tracking vendor just emailed a PDF labeled "Independent Bias Audit Summary." It has a logo, a methodology section, and a set of selection-rate tables. It also has a byline from a company you've never heard of — one that, it turns out, shares an office address with the vendor's compliance division.

That's the moment most HR Directors realize auditor selection isn't a formality. Local Law 144 doesn't just require an audit. It requires an audit performed by someone with no financial or employment stake in your organization or in the AEDT vendor whose tool is being tested. If the auditor and the vendor are connected — through ownership, contract, or a shared parent company — the audit doesn't satisfy the law, no matter how thorough the report looks.

This gets confusing fast, especially with vendors bundling "compliance packages" that include their own audit arm. By the end of this piece, you'll know exactly how to evaluate an independent bias auditor for Local Law 144, what questions expose a conflict before you sign anything, and how to document the decision so it holds up if anyone asks.

What Counts as an Independent Bias Auditor Under Local Law 144

Local Law 144 defines an automated employment decision tool (AEDT) as any computational process built on machine learning, statistical modeling, data analytics, or AI that produces a simplified output — a score, a classification, a ranking — used to substantially assist or replace discretionary hiring or promotion decisions (Perkins Coie, 2023). Once a tool meets that definition and is used to evaluate candidates or employees who reside in New York City, three obligations attach: an annual independent bias audit, a public summary of audit results posted with the AEDT's distribution date, and advance notice to candidates or employees at least 10 business days before the tool is used, along with an alternative-process option (Crowell & Moring LLP, 2023; Epstein Becker Green, 2023).

The law took effect January 1, 2023, and enforcement began July 5, 2023, after a delay from an originally announced April 15 date (Epstein Becker Green / Workforce Bulletin, 2023). Civil penalties run up to $500 for a first violation and $500–$1,500 for each subsequent one, accruing per violation per day — which means an unaudited AEDT left in use isn't a single fine, it's a clock (Office of the NY State Comptroller, 2025).

The word doing all the work in "independent bias auditor local law 144" is independent. The law and DCWP's rulemaking are specific: the auditor cannot have a current employment relationship with the employer, cannot have a financial relationship with the AEDT developer, and generally cannot have been involved in developing or deploying the tool being tested. That rules out a lot of arrangements that look like audits but aren't — a vendor's internal QA team, a consulting firm that also sells the employer other AI-governance services tied to the same tool, or an auditor whose fee structure depends on the employer renewing the AEDT contract.

If you're unsure whether a specific candidate auditor qualifies, DCWP is the authority to confirm it with — not a vendor's sales team, and not general internet guidance. This is an operations and documentation question you can prepare for; it becomes a legal determination the moment your organization's specific facts are in play, and that's outside what any workbook, checklist, or article can settle for you.

Auditor vs. Governance Platform: A Structural Distinction You Can't Skip

Not every company marketing "LL144 compliance" is offering the same thing, and mixing them up is the single most common auditor-selection mistake.

Some companies are structurally on the audit-execution side — they are independent auditors who perform the bias audit itself. Warden AI is one: it runs continuous bias-auditing engagements for HR-tech vendors and employers, and positions across several jurisdictions' AI-employment rules in addition to LL144. BABL AI is another — it performs LL144 bias audits under the ISAE 3000 assurance standard, the same framework used for other independent assurance engagements (BABL AI, 2025). Both are structured to actually issue the audit output the law requires.

Other companies are general AI-governance platforms — tools for tracking policies, risk registers, and model documentation across many regulatory frameworks (ISO 42001, NIST AI RMF, and others), without being built specifically around executing an LL144 bias audit. FairNow (now part of AuditBoard) and Holistic AI both sit in this category: broad governance tooling, not NY-statute-specific audit execution. VerifyWise occupies a middle position — it's a source-available, self-hostable GRC tool that covers LL144 among many other frameworks and separately offers auditor services, with a free self-hosted starting point that requires technical setup and custom enterprise deployment beyond that.

None of these three categories is "wrong" to exist. The mistake is treating a governance platform's dashboard as if it satisfies the audit requirement, or treating an audit-execution vendor's report as something you can also buy documentation tooling from for the same engagement — the independence requirement is exactly what prevents that second arrangement. A vendor that both audits your AEDT and sells you the paperwork trail around that same audit has a financial relationship with the outcome it's supposed to be testing.

WorkforceNewYork sits deliberately in a third lane: it doesn't perform, certify, or sign any bias audit, and it never scores an individual candidate or employee. The downloadable workbooks run the documentation and tracking operation around an audit that a separately engaged independent auditor performs — vendor status trackers, notice logs, audit calendars. That's a genuinely different function from either an audit-execution firm or a governance platform, and it's worth being precise about which one you're buying when you're comparing options. For a deeper side-by-side, see how a bias audit platform compares to an independent auditor.

Testing for Conflicts of Interest Before You Sign Anything

Before any auditor gets a signature, run a short independence test — the kind of thing DCWP would ask about if it ever reviewed your file.

Start with ownership and control. Does the proposed auditor have any equity relationship, parent-subsidiary tie, or reseller agreement with the AEDT vendor whose tool is being audited? Ask directly, in writing, and keep the answer.

Check the fee structure. An auditor whose compensation is contingent on the AEDT vendor's business relationship with you — a referral fee, a bundled-package discount tied to vendor renewal — creates exactly the financial entanglement the law is written to prevent.

Look at prior involvement. If the auditor's team helped design, train, or configure the AEDT being tested, that's disqualifying regardless of what the current contract says, because the independence requirement looks at the relationship to the tool, not just the paperwork around this year's engagement.

Confirm employment history. An auditor who is a former employee of the AEDT vendor, or of your organization, in a role connected to the tool, is a flag worth raising even if the formal employment relationship has ended.

None of this is exotic due diligence — it's the same category of question you'd ask before hiring an external financial auditor. The difference is that most HR teams have never had to ask it about an AI vendor relationship before, and vendors marketing "compliance bundles" don't always volunteer the answer. Our conflict-of-interest and independence guide walks through the specific relationships DCWP guidance flags, with a documentation template for recording what you asked and what you were told.

Running the RFP: Questions That Separate Auditors from Sales Decks

Once you've screened for conflicts, the RFP stage is where you separate a firm that can actually deliver a defensible audit from one that's repackaging a compliance checklist as an audit report.

Ask what standard or methodology governs the engagement — ISAE 3000, an internal framework, or something else — and ask them to show you a redacted sample report so you can see what the output actually looks like before you commit. Ask how they calculate selection rates and impact ratios, and whether they can walk you through a four-fifths (80%) rule calculation on request; that ratio — comparing a group's selection rate to the highest-selected group's rate, with anything below 80% flagged as a potential indicator of adverse impact — is the core arithmetic behind most LL144 audit outputs (via Assessment Systems, 2024, citing the EEOC Uniform Guidelines).

Ask what happens when historical applicant data has gaps — missing demographic fields, small sample sizes for certain job categories, or aggregation choices that could mask disparities. This matters more than it sounds: a 2025 academic review of published LL144 audits found that many reports may under-report disparities because of missing demographic data, opaque aggregation methods, and metrics that don't reflect how the tool is actually deployed day to day (ACM FAccT, "Auditing the Audits," 2025). An auditor who has a clear, specific answer for how they handle these gaps is a different kind of vendor than one who says "we follow best practices" and stops there.

Ask about turnaround time relative to your AEDT's distribution date, since the public summary has to be posted alongside that date, and ask what documentation they'll hand you at the end — not just a summary page, but underlying calculations you can reference if a candidate or a regulator asks a follow-up question. A full list of vetting questions, organized so you can score responses side by side, lives in our bias audit RFP question guide.

Local Law 144 doesn't require the cheapest audit or the fastest one. It requires one performed by an auditor with no financial or employment relationship to your organization or your AEDT vendor — and it puts the burden of proving that independence on you, not on the auditor's marketing page.

Scoping the Engagement So Nothing Falls Through the Cracks

A signed auditor doesn't automatically mean a complete audit. Scope disputes are where a lot of engagements quietly underdeliver: the auditor tests one AEDT when the employer is using three, or tests the vendor's default configuration when the employer customized the scoring weights, or audits hiring but not internal promotion even though the same tool scores both.

Before the engagement starts, put in writing exactly which tools are covered, which decision points (screening, ranking, interview scoring, promotion) are in scope, what data period the audit will draw from, and who is responsible for supplying the underlying applicant data — you, the AEDT vendor, or both. If your organization runs the tool across multiple job categories with meaningfully different applicant pools, confirm whether the audit will report those categories separately or roll them together, since rolled-up numbers can obscure a disparity that exists in one category and not another.

This is also where you settle deliverables: the public-facing summary DCWP requires is a short document, but you'll want the underlying calculations, methodology notes, and any raw output tables retained separately, since those are what you'd actually need if a candidate exercised the accommodation process or if a regulator followed up. Our engagement scope guide includes a scope-of-work template you can hand directly to a shortlisted auditor before signing.

Documenting the Choice — Your First Action Item

The enforcement record so far is a reason to take this seriously rather than treat it as a box to check. A December 2025 Comptroller's audit covering the July 2023–June 2025 period found NYC's LL144 enforcement "ineffective": DCWP identified only 1 of 32 reviewed companies as non-compliant, while the Comptroller's own auditors found 17 issues across those same companies — a gap that suggests agency review alone won't catch a bad auditor selection (Office of the NY State Comptroller, 2025). Separately, only two AEDT complaints were received during that entire audit window, and DCWP never investigated whether its own complaint-intake process was working (OSC, 2025); a 2024 academic study of 391 employers found only 18 had posted an audit report and only 13 had posted the required transparency notice at all (ACM FAccT, Wright & Muenster et al., 2024). None of that means enforcement stays light — it means the responsibility for getting auditor selection right currently sits almost entirely with you, not with a regulator who will catch a bad choice for you.

The single most useful thing you can do after reading this is write down, in one place, how you selected your auditor: the conflict-of-interest questions you asked, the RFP responses you compared, the scope you agreed to, and why you chose who you chose. That record is what turns "we hired an independent bias auditor for Local Law 144" from a claim into something you can actually show.

That's exactly what the Independent-Auditor Engagement RFP & Scoping Kit is built to hold — RFP question sets, an independence screening worksheet, and a scope-of-work template you can send straight to a shortlisted auditor. It doesn't perform the audit and it doesn't score anyone; it documents the selection process so the decision is defensible on its own. If you're still early in the process, start with our Local Law 144 compliance guide or the breakdown of who can perform a Local Law 144 bias audit, and if budget is the open question, our guide to what a bias audit costs in NYC covers what to expect before you get quotes.

Download the kit, run the independence test on your shortlist before the next audit cycle starts, and put the decision on paper while you're still making it — not after someone asks you to explain it.