Artificial intelligence has quietly reshaped recruitment. From sourcing, screening to selection of candidates, AI tools promise speed and scale in an ever-globalised workforce landscape following the boom of remote work infrastructure. However, a recent intervention by a UK regulator has surfaced regulatory pressure on the vendors building these systems.
The Information Commissioner’s Office (ICO) has audited certain providers of AI-driven recruitment tools in the UK and, published a thorough report effectively setting out a rulebook for the sector: AI in Recruitment Outcomes Report. The audits are a reminder that, even before the EU AI Act enters into force, UK data protection law is already imposing rigorous standards on how these systems are designed and deployed.
There has largely been an illusion of “upstream neutrality” where vendors offering HR tools have long assumed they have a safe regulatory distance from hiring decisions. The ICO’s findings make clear that this assumption is increasingly untenable, particularly where AI models are trained, calibrated, and deployed centrally across multiple clients.
The first point to note is the clarity in the ICO’s statement that where a provider develops a central AI model used across clients, it is likely to be acting as a controller, with full responsibility for how personal data is used in training and development. The audits found repeated attempts by providers to classify themselves as processors and push obligations downstream, often through vague or incomplete contracts.
This controller status triggers the requirement to conduct a Data Protection Impact Assessment (DPIA) early in development and update it as processing changes. The ICO expects DPIAs to contain a substantive analysis of the trade-offs between privacy and competing interests, including: accuracy versus explainability (more data points improve output accuracy but make it harder to explain how the AI works); data minimisation versus statistical validity (more personal information improves accuracy but conflicts with the minimisation principle); and transparency versus understandability (granular technical detail may seem more transparent but impacts comprehension). The ICO also found many DPIAs lacked a detailed map of data flows through the AI system and did not consider alternative approaches that might use less personal information to achieve the same outcomes.
Controller status also brings obligations around individual rights. Providers must document and implement processes to handle individual rights requests, including how each right will be managed within the AI tool. The ICO further expects controllers to maintain records of processing activities (RoPA) based on regular data flow mapping, recording purpose, lawful basis, additional conditions for special category data, and data sharing arrangements.
Controller status matters. It triggers the full weight of UK GDPR obligations: lawful basis, transparency and fairness. It also materially increases exposure in the event of regulatory scrutiny or private claims. Where responsibility for model design, training, and outputs sits with the vendor, liability is less easily displaced through contractual drafting alone. For many providers, this is a significant structural issue if an operating model and compliance programme rests on assumptions that no longer hold true in the UK regulatory landscape.
I will draw out some of the key themes from the ICO’s report in part 1 of this series spotlight.
Link to AI Bias: the problem providers thought they had solved AI Bias: the problem providers thought they had solved
There is extensive discussion in the recruitment industry on “debiasing” AI, but the ICO’s findings show some of this work is not yet fully effective.
A key concern is the widespread use of inferred demographic data, for example, predicting ethnicity or gender from names. The regulator found such data is not accurate enough to monitor bias and may be unlawful where used without a valid basis and without informing candidates.
Moreover, the regulator observed bias monitoring was often narrow in scope, limited to gender, ethnicity and age. Many providers were unable to assess other protected characteristics under the Equality Act as they could not be reliably inferred. The ICO also found that not all providers had used sampling techniques to ensure datasets were diverse and representative of the relevant population, highlighting the need to maintain clean datasets free of demographic proxies that could introduce bias.
The result is a compliance paradox: systems may be failing to detect the bias they introduce. Worse still, the ICO found that some tools included features allowing recruiters to filter candidates in or out of suggested lists based on inferred demographics, a functionality unlikely to be fair or have an appropriate lawful basis.
A failure to ensure potential bias in AI is properly detected and addressed may result in potential discrimination claims arising under the Equality Act. Candidates may assert that their particular protected characteristic was a reason for a role not being offered to them. If bias in the AI system is subsequently discovered which meant candidates with particular protected characteristics were not being progressed through the recruitment process, this could potentially create a liability for such a claim. It also may shift the burden of explanation onto the organisation deploying the tool.
The ICO outlines an alternative in collecting demographic data directly from candidates, for example via optional surveys. This may be more accurate, but it requires a clear lawful basis and purpose limitation analysis, particularly where data is repurposed for bias monitoring.
Link to AI Accuracy is not a sliding scale AI Accuracy is not a sliding scale
A striking regulatory comment in the report concerns accuracy. Some providers had adopted a pragmatic threshold: if an AI tool was “better than random”, it was deemed good enough. The ICO rejects this outright. Where AI materially influences recruitment decisions, “better than random” is unlikely to comply with the law. Systems must be tested rigorously and must reach defined accuracy thresholds before processing personal data. From a risk perspective, organisations should also consider how those thresholds are evidenced and documented, particularly where outputs may later need to be defended in the context of a complaint, audit or claim.
Critically, the ICO also recommended that providers keep training and testing data separate. Where AI is trained with information and then tested with the same data, accuracy or bias issues may remain undetected until after launch. The regulator further suggests engaging cognitive behavioural and psychometric experts to regularly test and review AI logic, scoring, and outputs for potential accuracy or bias issues.
Link to Transparency: privacy notices criticised Transparency: privacy notices criticised
The ICO found many providers relying on generic, multi-purpose privacy policies, often covering multiple jurisdictions and processing activities. These were held to be insufficient.
The regulator’s expectation is clear:
- Have a processing-specific candidate privacy notices
- Include a clear explanation of AI logic, outputs and training uses
- Disclose the lawful basis and retention periods
- Supplement text-based privacy information with informative pop-up messages, bite-sized information at the point of processing, or visual aids such as data flow maps
Of particular note is the ICO’s stance on Article 14(5)(b), the “disproportionate effort” exemption. Several providers relied on this exemption when scraping data from public sources such as job platforms. The ICO was not persuaded, signalling a materially narrower interpretation of the exemption in practice. This has direct implications for large-scale data sourcing strategies which rely on passive transparency and may now carry increased regulatory risk if not revisited.
Where providers held names and contact details, they could not justify why notifying individuals would be disproportionate. Simply publishing a website privacy policy was not enough. Individuals must be actively informed, typically within one month of collection. This is a significant tightening of practice and large-scale sourcing models built on passive transparency should be reviewed.
Link to Human oversight: present, but not always meaningful Human oversight: present, but not always meaningful
AI vendors routinely emphasise “human-in-the-loop” safeguards. The ICO’s findings suggest these controls are often superficial in practice.
Most tools were designed to support, rather than replace, human decision-making. Yet:
- human review processes were not always formalised and reviewers were not consistently trained
- feedback loops were weak or undocumented i.e. in some cases, providers relied on recruiters to flag errors but without clear mechanisms to do so, or evidence that issues were addressed.
The result is a familiar compliance gap: human oversight exists in theory, but often operates as a control of form rather than substance, providing limited mitigation in practice. Such a gap in a recruitment process may have several unintended consequences and therefore risks. For example, there may be a failure to identify and support candidates with disabilities who require certain adjustments to be made in order to remove any disadvantage they may be experiencing in the recruitment process.
Link to Data hunger and purpose creep Data hunger and purpose creep
AI systems require large volumes of data. In parallel, data protection law constrains that with data minimisation a core UK GDPR principle.
The ICO found extensive use of scraped data from public profiles, often combined at scale and reused for training and product development. In many cases, providers could not demonstrate that these secondary uses were compatible with the original purpose of collection. The regulator was particularly critical where providers had no contracts with source sites confirming information was collected lawfully, and could not show the new use was compatible with the original purpose.
Retention practices were equally problematic. Some providers maintained candidate databases indefinitely “just in case”, or reset retention periods with each update effectively creating permanent records.
Against this, the ICO emphasises a more restrictive model:
- define a minimum data set (often limited to core CV data)
- use pseudonymisation or aggregated data where possible
- adopt techniques such as k-fold cross-validation to reduce data volume without sacrificing accuracy
- limit collection to defined parameters, for example up to 10 years of job experience, and collect demographic information optionally and exclusively for bias monitoring purposes
If a provider’s growth strategy is built on accumulating ever-larger datasets, this challenge should be noted.
Providers should also note that where they rely on legitimate interests as a lawful basis, a legitimate interests assessment (LIA) must be conducted and documented which will consider some of these issues.
A related and emerging issue is the downstream impact of AI-assisted working practices within organisations themselves. In enforcement and contentious contexts, we are increasingly seeing internal use of generative AI tools giving rise to large volumes of additional personal data, including inferred or speculative content, which must then be captured and reviewed in response to data subject access requests. This creates both operational strain and a heightened risk profile, particularly where the content reflects inappropriate or unverified assumptions about individuals. Organisations should ensure that internal AI usage policies and training are aligned with their data protection obligations in this regard.
Link to Security: complex AI systems, expanding risks Security: complex AI systems, expanding risks
While the audits found generally strong technical controls, encryption, monitoring and segregated environments, the report also highlights the increasing complexity of AI systems. These tools involve multiple integrations, cloud infrastructures and third-party processors, creating a broader attack surface and more complex data flows.
At the same time, weaknesses persisted: (a) inconsistent access management, (b) insufficient review of audit logs; and (c) unclear processes for internal role changes. The lesson is that AI security risks are not wholly “novel”, but they are materially amplified by scale, integration, and data sensitivity, increasing both the likelihood and impact of failures across interconnected systems. In particular, reliance on third-party integrations and evolving model architectures can make root cause analysis and incident containment more complex in practice and therefore more prone to error. This should be explored in detail in the DPIA.
Link to The coming convergence The coming convergence
Overall, the ICO’s audit reads like preparatory enforcement if the expectations are not met. The message for providers is that compliance cannot be bolted on. It must be designed into AI systems from the outset, spanning data sourcing, model training, governance and user interaction. Some organisations are taking a “wait and see” approach to high-risk AI systems in the recruitment space caught by the EU AI Act in light of the Omnibus proposals. However, the UK intervention report is grounded in current data protection principles which already apply.
For providers, the ICO’s findings translate into a set of immediate, actionable priorities: review controller/processor classification with legal counsel; conduct or update DPIAs with documented trade-off analysis and detailed data flow mapping; separate training and testing datasets; engage external experts (psychometric, cognitive behavioural) for ongoing validation; revise privacy notices to be UK-specific and candidate-focused with layered transparency mechanisms; conduct LIAs where relying on legitimate interests; implement processes for handling individual rights requests and maintain a comprehensive RoPA; review data scraping practices and purpose compatibility; implement defined retention limits; and test breach response processes through periodic desktop simulations.
Part two will examine how the EU AI Act sharpens these obligations, and what it means for providers operating across both regimes – in addition to what employers/deployers should consider when using this tools from a contractual and compliance perspective.