In a series of votes in the European Parliament this morning MEPs have backed a raft of amendments to the bloc’s draft AI legislation — including agreeing a set of requirements for so called foundational models which underpin generative AI technologies like OpenAI’s ChatGPT.

The text of the amendment agreed by MEPs in two committees put obligations on providers of foundational models to apply safety checks, data governance measures and risk mitigations prior to putting their models on the market — including obligating them to consider “foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law”.

The amendment also commits foundational model makers to reduce the energy consumption and resource use of their systems and register their systems in an EU database set to be established by the AI Act. While providers of generative AI technologies (such as ChatGPT) are obliged comply with transparency obligations in the regulation (ensuring users are informed the content was machine generated); apply “adequate safeguards” in relation to content their systems generate; and provide a summary of any copyrighted materials used to train their AIs.

In recent weeks MEPs have been focused on ensuring general purpose AI will not escape regulatory requirements, as we reported earlier.

Other key areas of debate for parliamentarians included biometric surveillance — where MEPs also agreed to changes aimed at beefing up protections for fundamental rights.

The lawmakers are working towards agreeing the parliament’s negotiating mandate for the AI Act to unlock the next stage of the EU’s co-legislative process.

MEPs in two committees, the Internal Market Committee and the Civil Liberties Committee, voted on some 3,000 amendments today — adopting a draft mandate on the planned artificial intelligence rulebook with 84 votes in favour, 7 against and 12 abstentions.

“In their amendments to the Commission’s proposal, MEPs aim to ensure that AI systems are overseen by people, are safe, transparent, traceable, non-discriminatory, and environmentally friendly. They also want to have a uniform definition for AI designed to be technology-neutral, so that it can apply to the AI systems of today and tomorrow,” the parliament said in a press release.

Among the key amendments agreed by the committees today are an expansion of the list of prohibited practices — adding bans on “intrusive” and “discriminatory” uses of AI systems such as:

• “Real-time” remote biometric identification systems in publicly accessible spaces;
• “Post” remote biometric identification systems, with the only exception of law enforcement for the prosecution of serious crimes and only after judicial authorization;
• Biometric categorisation systems using sensitive characteristics (e.g. gender, race, ethnicity, citizenship status, religion, political orientation);
• Predictive policing systems (based on profiling, location or past criminal behaviour);
• Emotion recognition systems in law enforcement, border management, workplace, and educational institutions; and
• Indiscriminate scraping of biometric data from social media or CCTV footage to create facial recognition databases (violating human rights and right to privacy).

The latter, which would outright ban the business model of the controversial US AI company Clearview AI comes a day after France’s data protection watchdog hit the startup with another fine for failing to comply with existing EU laws. So there’s no doubt enforcement of such prohibitions on foreign entities that opt to flout the bloc’s rules will remain a challenge. But the first step is to have hard law.

Commenting after the vote in a statement, co-rapporteur and MEP Dragos Tudorache, added:

Given the profound transformative impact AI will have on our societies and economies, the AI Act is very likely the most important piece of legislation in this mandate. It’s the first piece of legislation of this kind worldwide, which means that the EU can lead the way in making AI human-centric, trustworthy and safe. We have worked to support AI innovation in Europe and to give start-ups, SMEs and industry space to grow and innovate, while protecting fundamental rights, strengthening democratic oversight and ensuring a mature system of AI governance and enforcement.

A plenary vote in parliament to seal the mandate is expected next month (during the 12-15 June session), after which trilogue talks will kick off with the Council toward agreeing a final compromise on the file.

Back in 2021, when the Commission’ presented its draft proposal for the AI Act it suggested the risk-based framework would create a blueprint for “human” and “trustworthy” AI. However concerns were quickly raised that the plan fell far short of the mark — including in areas related to biometric surveillance, with the Commission only proposing a limited ban on use of highly intrusive technology like facial recognition in public.

Civil society groups and EU bodies pressed for amendments to bolster protections for fundamental rights — with the European Data Protection Supervisor and European Data Protection Board among those calling for the legislation to go further and urging EU lawmakers to put a total ban on biometrics surveillance in public.

MEPs appear to have largely heeded civil society’s call. Although concerns do remain. (And of course it remains to be seen how the proposal MEPs have strengthened could get watered back down again as Member States governments enter the negotiations in the coming months.)

Other changes parliamentarians agreed in today’s committee votes include expansions to the regulation’s (fixed) classification of “high-risk” areas — to include harm to people’s health, safety, fundamental rights and the environment.

AI systems used to influence voters in political campaigns and those used in recommender systems by larger social media platforms (with more than 45 million users, aligning with the VLOPs classification in the Digital Services Act), were also put on the high-risk list.

At the same time, though, MEPs backed changes to what counts as high risk — proposing to leave it up to AI developers to decide if their system is significant enough to meet the bar where obligations applying, something digital rights groups are warning (see below) is “a major red flag” for enforcing the rules.

Elsewhere, MEPs backed amendments aimed at boosting citizens’ right to file complaints about AI systems and receive explanations of decisions based on high-risk AI systems that “significantly” impact their rights.

The lack of meaningful redress for individuals affected by harmful AIs was a major loophole raised by civil society groups in a major call for revisions in fall 2021 who pointed out the glaring difference between the Commission’s AI Act proposal and the bloc’s General Data Protection Act, under which individuals can complain to regulators and pursue other forms of redress.

Another change MEPs agree on today is a reformed role for body called the EU AI Office, which they want to monitor how the rulebook is implemented — to supplement decentralized oversight of the regulation at the Member State level.

While, in a nod to the perennial industry cry that too much regulation is harmful for “innovation”, they also added exemptions to rules for research activities and AI components provided under open-source licenses, while noting the law promotes regulatory sandboxes, or controlled environments, being established by public authorities to test AI before its deployment.

Digital rights group EDRi, which has been urging major revisions to the Commission draft, said everything it had been pushing for was passed by MEPs “in some form or another” — flagging particularly the (now) full ban on facial recognition in public; along with (new) bans on predictive policing, emotion recognition and on other harmful uses of AI.

Another key win it points to is the inclusion of accountability and transparency obligations on deployers of high risk AI — applying on them a duty to do a fundamental rights impact assessment and mechanisms by which people affected can challenge AI systems.

“The Parliament is sending a clear message to governments and AI developers with its list of bans, ceding civil society’s demands that some uses of AI are just too harmful to be allowed, Sarah Chander, EDRi senior policy advisor,” told TechCrunch.

“This new text is a vast improvement from the Commission’s original proposal when it comes to reigning in the abuse of sensitive data about our faces, bodies, and identities,” added, Ella Jakubowska, an EDRi senior policy advisor who has focused on biometrics. 

However EDRi said there are still areas of concern — pointing to use of AI for migration control as one big one.

On this, Chander noted that MEPs failed to include in the list of prohibited practices where AI is used to facilitate “illegal pushbacks”, or to profile people in a discriminatory manner — which is something EDRi had called for. “Unfortunately, the [European Parliament’s] support for peoples’ rights stops short of protecting migrants from AI harms, including where AI is used to facilitate pushbacks,” she said, suggesting: “Without these prohibitions the European Parliament is opening the door for a  panopticon at the EU border.”

The group said it would also like to see improvements to the proposed ban on predictive policing — to cover location based predictive policing which Chander described as “essentially a form of automated racial profiling”. She said it’s worried that the proposed remote biometrics identification ban won’t cover the full extent of mass surveillance practices it’s seen being used across Europe.

“Whilst the Parliament’s approach is very comprehensive [on biometrics], there are a few practices that we would like to see even further restricted. Whilst there is a ban on retrospective public facial recognition, it contains an exception for law enforcement use which we still consider to be too risky. In particular, it could incentivise mass retention of CCTV footage and biometric data, which we would clearly oppose,” added Jakubowska, saying it would also want to see the EU outlaw emotion recognition no matter the context — “as this ‘technology’ is fundamentally flawed, unscientific, and discriminatory by design”.

Another concern EDRi flags is MEPs’ proposal to let AI developers judge if their systems are high risk or not — as this risk undermining enforceability.

“Unfortunately, the Parliament is proposing some very worrying changes relating to what counts as ‘high-risk AI. With the changes in the text, developers will be able to decide if their system is ‘significant’ enough to be considered high risk, a major red flag for the enforcement of this legislation,” Chander suggested.

While today’s committee vote is a big step towards setting the parliament’s mandate — and setting the tone for the upcoming trilogue talks with the Council — much could still change and there is likely to be some pushback from Member States governments, which tend to be more focused on national security considerations than caring for fundamental rights.

Asked whether it’s expecting the Council to try to unpick some of the expanded protections against biometric surveillance Jakubowska said: “We can see from the Council’s general approach last year that they want to water down the already insufficient protections in the Commission’s original text. Despite having no credible evidence of effectiveness — and lots of evidence of the harms — we see that many member state governments are keen to retain the ability to conduct biometric mass surveillance.

“They often do this under the pretence of ‘national security’ such as in the case of the French Olympics and Paralympics, and/or as part of broader trends criminalising migration and other minoritised communities. That being said, we saw what could be considered ‘dissenting opinions’ from both Austria and Germany, who both favour stronger protections of biometric data in the AI Act. And we’ve heard rumours that several other countries are willing to make compromises in the direction of the biometrics provisions. This gives us hope that there will be a positive outcome from the trilogues, even though we of course expect a strong push back from several Member States.”

Giving another early assessment from civil society, Kris Shrishak, a senior fellow at the Irish Council for Civil Liberties (ICCL), which also joined the 2021 call for major revisions to the AI Act, also cautioned over enforcement challenges — warning that while the parliament has strengthened enforceability by amendments that explicitly allow regulators to perform remote inspections, he suggested MEPs are simultaneously tying regulators hands by preventing them access to source code of AI systems for investigations.

“We are also concerned that we will see a repeat of GDPR-like enforcement problems,” he told TechCrunch.

On the plus side he said MEPs have taken a step towards addressing “the shortcomings” of the Commission’s definition of AI systems — notably with generative AI systems being brought in scope and the application of transparency obligations on them, which he dubbed “a key step towards addressing their harms”.

But — on the issue of copyright and AI training data — Shrishak was critical of the lack of a”firm stand” by MEPs to stop data mining giants from ingesting information for free, including copyright-protected data.

The copyright amendment only requires companies to provide a summary of copyright-protected data used for training — suggesting it will be left up to rights holders to sue.

Asked about possible concerns that exemptions for research activities and AI components provided under open source licenses might create fresh loopholes for AI giants to escape the rules, he agreed that’s a worry.

“Research is a loophole that is carried over from the scope of the regulation. This is likely to be exploited by companies,” he suggested. “In the context of AI it is a big loophole considering large parts of the research is taking place in companies. We already see Google saying they are ‘experimenting’ with Bard. Further to this, I expect some companies to claim that they develop AI components and not AI systems (I already heard this from one large corporation during discussions on General purpose AI. This was one of their arguments for why GPAI [general purpose AI] should not be regulated).”

EU lawmakers back transparency and safety rules for generative AI by Natasha Lomas originally published on TechCrunch