Last week, a European Data Protection Authority imposed a substantial fine for a company’s use of an algorithmic system that automatically restricted individuals’ access to income-generating opportunities via an online platform and the failure to provide adequate transparency. Accounts were deactivated where the system detected suspected fraud or low customer ratings and no human assessment took place. According to the authority, this constitutes prohibited automatic decision making within the meaning of Art. 22 GDPR, which states that a “data subject shall have the right not to be subject to a decision based on automated processing, including profiling, which produces legal effects concerning him or her or similarly significant affects him or her”.
The investigation followed complaints from 171 individuals. The fine was calculated by reference to the company’s worldwide annual turnover. The decision is not yet final and is subject to appeal.
The case is not merely a platform-economy issue. Automated scores and system-generated flags increasingly influence everyday business decisions, including decisions affecting employees and customers. While many automated processes are routine and legally unproblematic, Art. 22 GDPR may apply where a decision is based solely on automated processing and produces legal or similar significant effects. Examples include loss of income, as in the decision at hand, denial of credit or insurance, rejection of a job application, or exclusion from an important service.
Where automated-decision making is permitted under one of the limited exceptions in Art. 22 GDPR, businesses must implement appropriate safeguards. These include meaningful human intervention, transparent information about the decision-making process and an effective opportunity to contest the decision. A nominal review or a routine approval of a system generated recommendation is unlikely to be sufficient.
Relevant use cases may include, for example, automated recruitment screening, allocation or withdrawal of shifts, performance scoring, fraud detection, refusal of returns or payment options, as well as automated access restrictions. Business should identify where automated outputs trigger adverse consequences and assess whether the decision is genuinely reviewed by a person with sufficient information, authority and ability to depart from the system’s recommendation.
While the decision concerns the GDPR and does not determine whether the system in question qualifies as an AI system under the AI Act, the decision may nevertheless provide useful context for the AI Act’s human oversight requirements. In the employment context, AI-supported recruitment, performance evaluation, task allocation or disciplinary decisions may qualify as high-risk under Annex III. This may be the case even where a human formally makes the final decision. Effective human oversight under the AI Act must therefore be substantive rather than merely procedural: the responsible person must be able to understand and critically assess the system’s output and, where appropriate, disregard, override or reverse it.
Businesses should review their inventories of automated and AI-supported systems, conduct or update a data protection impact assessment where required, update their privacy information where necessary, establish effective procedures for dealing with objections to automated decisions and document both the decision-making process and any human review. Although the AI Act requirements for Annex III high risk systems will not apply until 2 December 2027, the necessary governance and documentation should be developed now.
Disclaimer: While every effort has been made to ensure that the information contained in this article is accurate, neither its authors nor Squire Patton Boggs accepts responsibility for any errors or omissions. The content of this article is for general information only, and is not intended to constitute or be relied upon as legal advice.