Historical Data vs. Test Data in a Bias Audit

Your auditor emails back one question: "Do you have real hiring data, or should we build a test set?"

You expected the independent auditor to ask for your AEDT's technical documentation. You did not expect them to ask about your last two years of hiring data first. That question — historical or test — is the fork in the road that decides what you're about to spend the next few weeks assembling. Hand over the wrong thing, or hand over a partial version of the right thing, and the audit stalls, or worse, gets built on a foundation that doesn't reflect how your tool actually performs on your actual candidates.

This confusion is common. Employers deploying an automated employment decision tool (AEDT) for the first time often assume "the auditor will handle it" — and the auditor will, but only once you've told them, accurately, what data actually exists in your applicant tracking system. By the end of this piece you'll know which data type your audit is likely to draw on, what each path requires you to prepare, and how to hand your auditor a package that doesn't bounce back with follow-up questions.

What "historical data" means in a bias audit request

Historical data is exactly what it sounds like: the real, already-recorded outcomes from actual candidates or employees who went through your AEDT during actual use. Under Local Law 144, the preferred basis for a bias audit is this kind of real-world usage data — actual selection rates, actual scoring outcomes, actual demographic breakdowns of who applied and who advanced, run through the tool as it was genuinely deployed.

The advantage is obvious: historical data reflects how the tool behaves on your candidate pool, in your hiring context, with your job requisitions. A test set can approximate a distribution. Historical data is the distribution.

The catch is that historical data is only useful to an auditor if it's complete enough to compute something meaningful. That generally means enough records, across enough of the demographic categories the audit needs to measure, tied cleanly to which candidates the AEDT actually processed. If your applicant tracking system exported half the field, or demographic self-identification was optional and mostly skipped, or the AEDT was swapped mid-year and the old vendor's data lives in a system nobody has login credentials for anymore — the audit hits a wall that has nothing to do with the vendor's algorithm and everything to do with your recordkeeping.

We don't have a single published sufficiency threshold to hand you here — how much historical data counts as "enough" is a call your engaged independent auditor makes based on your applicant volume and the categories being measured. Confirm the specific standard your auditor is applying before you assume last year's export is adequate.

When auditors turn to test data instead

If historical data doesn't exist yet — a newly deployed AEDT, a tool used for the first time this hiring cycle, or a role category too small to generate a usable sample — the audit can shift to test data: a constructed dataset, built and run through the tool specifically to produce the metrics the audit needs.

Test data is a fallback with a real function, not a shortcut. It lets an employer get a first bias audit completed before the tool has accumulated enough real usage to analyze, which matters given that Local Law 144 requires an audit before an AEDT already in production has gone a full year without one, on top of the annual obligation to keep testing (Crowell & Moring LLP, 2023; Epstein Becker Green, 2023). Waiting a year to accumulate "enough" historical data is not a compliant option if the tool is already live and unaudited.

The tradeoff is fidelity. A constructed test set is only as representative as whoever built it made it. If it doesn't reflect your real applicant demographics, real job requirements, or real scoring conditions, the resulting audit measures the tool's behavior on a hypothetical population — useful, but a step removed from how the tool actually treats your candidates. This is exactly the kind of documentation-quality question your auditor will ask you to help answer, and exactly the kind of gap outside reviewers have flagged in published LL144 audits generally: research on published audit summaries has found that many under-report disparities because of missing demographic data, opaque aggregation, or metrics that don't reflect real deployment conditions (ACM FAccT, 2025).

The four-fifths rule reads the same way regardless of data source

Whichever data type feeds the audit, the underlying math an auditor is typically checking runs through some version of the four-fifths rule: a selection rate for any group that falls below 80% of the rate for the highest-selected group is a signal of possible adverse impact (EEOC Uniform Guidelines, via Assessment Systems, 2024).

Here's a worked example, not a real audit finding, just the arithmetic: if your highest-selected group advances at a 50% rate, 80% of that is 40%. Any group advancing below a 40% rate on this measure would flag for a closer look. Feed that formula real historical selection rates and you're measuring your actual hiring pipeline. Feed it a constructed test set and you're measuring the tool's behavior on a simulated one. The formula doesn't change — the confidence you can place in what it tells you does.

What your data-provision packet needs, either path

Whether your auditor ends up working from historical data vs. test data, the packet you assemble tends to need the same skeleton:

• A clear record of which AEDT version processed which candidates and when
• Demographic category data captured at the point of application, not reconstructed after the fact
• A defensible explanation for any gaps — missing fields, low sample sizes, vendor transitions — rather than silence
• Documentation of which job categories or requisitions the AEDT actually touched

None of this is legal advice, and preparing it isn't the same as performing the audit — a WorkforceNewYork workbook helps you organize and hand off this data; it does not conduct, certify, or sign the bias audit itself. That determination sits entirely with your engaged independent auditor, and any question about whether your specific dataset meets the statutory bar belongs with DCWP or your own counsel, not with a downloadable template.

The audit measures the tool. Your data-provision packet decides whether it's measuring the tool against reality or against an approximation of reality.

Why this keeps tripping employers up in practice

This isn't a hypothetical gap. Outside review of the compliance landscape has found real shortfalls at scale: an academic study of 391 employers subject to LL144 found only 18 had posted audit results publicly and only 13 had posted the required transparency notice (ACM FAccT, Wright & Muenster et al., 2024). A lot of that gap traces back to exactly this stage — employers who didn't know what data their auditor needed, didn't have it organized, and let the process stall rather than push through it. A New York State Comptroller review covering the enforcement period found the agency's own compliance checks caught a fraction of the non-compliance its own auditors later found on the same set of companies (OSC, 2025) — a reminder that "the auditor will catch it" is not a substitute for arriving with clean data in the first place.

What to prepare now

Before your next conversation with your auditor, know which data type your engagement is likely to use, and get ahead of the gaps historical data provision typically exposes: incomplete demographic fields, vendor-transition blind spots, and job-category coverage that doesn't match what the AEDT actually processed. The AEDT Data-Provision & Historical-Data Prep Workbook is built to walk through exactly that inventory before your auditor asks for it.

For the fuller picture of what a bias auditor typically requests beyond the data itself, see what a bias auditor needs and how to scope an independent auditor's engagement under Local Law 144. If you're preparing to interpret the results once the audit is back, how to read a bias audit summary walks through that next step, and the Local Law 144 compliance guide covers the full obligation end to end.