Artificial Intelligence Compliance: The Next Big Challenge for Risk Managers.

Artificial Intelligence (‘AI’) is in the news almost every day – often featuring scare stories, perpetuated by the likes of Elon Musk in his interview with Rishi Sunak, during the Global AI Safety Summit in London during November 2023. Musk then said that AI will be “the most disruptive force in history… AI can do everything. I don’t know if that makes people comfortable or uncomfortable. It’s both good and bad”. Our suggestion in this blog is that Risk Managers should be starting to feel uncomfortable about the challenge of managing this emerging, serious, and complex new risk which is coming their way.

The FCA Handbook requires the risk management function to ensure that all material risks are identified, measured, and properly reported. It must also be actively involved in elaborating the firm’s risk strategy and in all material risk management decisions. In terms of reporting, it must be able to deliver a complete view of the whole range of risks to the firm. In our view, this places AI risk management and compliance squarely on the desk of the Risk Director or Chief Risk Officer.

So, what should Risk Managers be doing now to ensure that the deployment of AI in their firms does not create reputational risk to the firm or deliver poor outcomes for consumers? We consider the European AI Act, which is the first formalised AI regulation to be approved, as well as the recent UK government guidance on AI Assurance . We then suggest some proactive steps that risk managers can be taking now to keep ahead of the game.

The EU AI Act – Introduction.

On the 13th of March 2024, the European Parliament approved the Artificial Intelligence Act (‘EUAIA’), which establishes obligations for firms with regards to their development and use of AI systems. These obligations are based upon the potential risks and level of impact of the AI system.

The EUAIA is a significant milestone in the years-long push for ethical and responsible use of AI. It is also a regulation with teeth and is part of a growing movement to protect society from the risks that a range of emerging technologies present. Importantly, compliance with the EUAIA should also be regarded as a way for companies to protect their own brands, performance, and reputations.

The EUAIA obliges firms to risk assess all of their systems that deploy AI, throughout the AI lifecycle, with the consequent risk assessment driving a range of compliance obligations.

Other key features of the EUAIA include:

Safeguards on general purpose artificial intelligence.
Limits on the use of biometric identification systems by law enforcement.
Bans on social scoring and AI used to manipulate or exploit user vulnerabilities.
Rights for consumers to launch complaints and receive meaningful explanations.

In many respects the EUAIA is very similar to Europe’s General Data Protection Regulation (‘GDPR’), passed in 2016, but for AI, and it is widely expected to have a similar impact. Like the GDPR, the EUAIA contains some draconian penalties for non-compliance (with fines ranging up to EUR 35M or 7% of global annual turnover).

The EUAIA mandates the establishment of a risk management system for high-risk AI systems, emphasizing a continuous, iterative process throughout the AI system's lifecycle. It also requires providers of high-risk AI systems to implement a quality management system, documenting policies, procedures, and instructions covering various aspects, including regulatory compliance, design and development, data management, and risk management.
The Act stresses the importance of human oversight for high-risk AI systems, ensuring that decisions made by, or with the assistance of, AI can be overseen, interpreted, and, if necessary, overridden by humans. This is crucial for preventing automation bias and ensuring that AI systems' outputs are aligned with ethical and safety standards.
Providers are required to ensure transparency and provide adequate information to deployers and users, enabling them to use the AI system in compliance with its intended purpose and in a manner that respects fundamental rights.
The establishment of new administrative infrastructures, such as the AI Office, a scientific panel, an AI Board, and an advisory forum, is crucial for overseeing the implementation of the EUAIA and providing technical expertise and advisory support to the Commission.

Impact of the EUAIA outside of the EU.

The EUAIA will apply primarily to providers of AI systems established within the EU, or in a third country which either place AI systems on the EU market or puts them into service in the EU, as well as to users of AI systems located in the EU. To prevent circumvention of the regulation, the new rules will also apply to providers and users of AI systems located in a third country where the output produced by those systems is used in the EU. Unless firms can be satisfied that their AI systems will not, under any circumstances, have any EU nexus, they will need to follow the rules.

There are several exceptions and specific conditions under which the EUAIA does not apply including:

Public Authorities and International Organisations.
Military, Defence, or National Security Purposes.
Scientific Research and Development.
Personal, Non-Professional Activity.
More Favourable Laws for Workers: The regulation does not preclude the Union or Member States from maintaining or introducing laws, regulations, or administrative provisions more favourable to workers concerning the use of AI systems by employers.

What do firms have to do to comply with the EUAIA?

The EUAIA contains different rules for four different risk levels posed by AI systems. The risk levels are: ‘Unacceptable Risk’, ‘High Risk’, ‘General purpose and generative AI’ and ‘Limited Risk’. The EUAIA requires businesses to identify, and risk classify their AI systems into these risk levels, a process referred to as ‘Know Your AI System’ (‘KYAIS’).

Depending on the number of AI systems being deployed in a firm, KYAIS, represented in figure 1 below, could well be a very onerous process.

Figure 1: A Typical KYAIS Process

The higher the risk classification of an AI system, the more onerous the specific obligations in terms of compliance and potential remediation.

To start your firm’s KYAIS approach, we recommend that you follow a three-step-approach.

Identify AI Systems - Start by identifying the AI systems, your organisation uses, especially those falling under the EUAIA radar. Keep a close eye on the high-risk ones, like safety critical systems or standalone products that need third-party assessments.
Asses AI Risks - Once you've found them, it's time to assess the risk level for each AI system, following the criteria laid out in the EUAIA. You'll want to separate the high-risk ones from the rest. Annex III of the Act breaks down classifications and real-world examples, making it easier to understand.
Get a Second Opinion - It's always a good idea to get a fresh perspective, especially when it comes to complex and novel legal issues. Consider reaching out to law firms or other experts to double-check your risk classifications. They'll evaluate your AI systems based on Articles 5-6, Annex III of the EIAIA, and Commission guidelines, giving you advice on what steps to take to stay compliant.

UK Government Guidance on ‘AI Assurance’.

The UK Government is taking a ‘pro-innovation approach’ to AI regulation and has consulted widely on their proposals to regulate AI. They have just published the response to their 2023 consultation exercise. They sum up their current ‘wait and see’ position as follows:

"As AI systems advance in capability and societal impact, it is clear that some mandatory measures will ultimately be required across all jurisdictions to address potential AI-related harms, ensure public safety, and let us realise the transformative opportunities that the technology offers. However, acting before we properly understand the risks and appropriate mitigations would harm our ability to benefit from technological progress while leaving us unable to adapt quickly to emerging risks. We are going to take our time to get this right – we will legislate when we are confident that it is the right thing to do. a pro-innovation approach to AI regulation: government response to consultation February 2024."

In February 2024, the Department for Science, Innovation & Technology (‘DSIT’) published guidance to introduce the concept of ‘AI Assurance’. In a helpful document they explain the concept and what types of measures they would expect those building, acquiring, or employing AI systems to take to deliver the government’s five cross-sectoral principles and deliver good AI outcomes. DSIT says:

“AI assurance is consequently a crucial component of wider organisational risk management frameworks for developing, procuring, and deploying AI systems, as well as demonstrating compliance with existing - and any relevant future – regulation”.

AI Assurance will:

Help to build confidence in AI systems by measuring and evaluating reliable, standardised, and accessible evidence about the capabilities of these systems.
Measure whether AI systems will work as intended, hold limitations, or pose risks.
Determine how those risks are being mitigated to ensure that ethical considerations are built-in throughout the AI development lifecycle.

The guidance, which mirrors many of the measures in the EUAIA, goes on to identify the ‘AI assurance toolkit’ based on:

1. Measurement.

Gathering qualitative and quantitative data on how AI systems function, to ensure that they perform as intended. This might include information about performance, functionality, and potential impacts in different contexts.

2. Evaluation.

Techniques to assess the risks and impacts of AI systems and inform further decision-making. This might include evaluating the implications of an AI system against agreed benchmarks set out in standards and regulatory guidelines.

3. Communication.

A range of communication techniques can be applied to ensure effective communication both within an organisation and externally. This might include collating findings into reports or presenting information in a dashboard, as well as external communication to the public to set out steps an organisation has taken to assure their AI systems.
The guidance also describes a range of AI assurance mechanisms which will be familiar to risk managers, including risk and impact assessments, bias and compliance audits, conformity assessments and formal verification.

What should Risk Managers be doing now?

So, we have the EUAIA, which will undoubtedly apply to many UK-based firms, and the DSIT ‘guidance’ which will evolve over time. Financial services firms also have to consider other regulations which are not AI specific, together with the imperative to ensure the confidence and trust of their customers.

We would expect Senior Management (IT, Risk) to be aware of the firm’s existing AI applications and maybe also those in the pipeline. Some firms may have already established a mechanism to consider potential AI impacts on customer outcomes. We would contend that regulated firms are already obliged do this to comply with the FCA’s Consumer Duty. This regulation stipulates that firms must design products and services that aim to secure good consumer outcomes. And they must demonstrate how all parts of their supply chain – from sales to after-sales, distribution, and digital infrastructure – deliver these outcomes.
The FCA’s Senior Managers & Certification Regime also provides a framework to respond to innovations in AI. This makes clear that senior managers are ultimately accountable for the activities of the firm, and the risks associated with those activities, including AI risks. However, in many firms, the specifics of the EUAIA, its application and impacts may not yet have been identified and considered by Senior Management.
An effective response to the EUAIA will be an enterprise-wide initiative, involving a wide range of stakeholders whose support will be critical. Board and Executive engagement will be critical to success. The Board is ultimately responsible for protecting the organisation from ethical, reputational, and regulatory risks. They are also responsible for identifying the relative importance of other kinds of strategic priorities, including the type and velocity of innovation, the pace and appropriateness of mergers and acquisitions and other budgeting priorities. Senior Management will need to commit to an appropriate investment in skilled resources to develop and run an effective AI Compliance programme.
Even whilst the AI Compliance programme is still developing, the firm should be looking to develop its understanding of AI assurance and anticipating likely future requirements.Data scientists and engineers, as well as the business units that employ them, should be made aware of the AI Compliance requirements and the firm’s governance and oversight of AI risks. If they know what they can and can’t do now, and in the future, this can be factored into the firm’s AI strategy.
In our view, ownership and delivery of the AI Compliance programme should ultimately lie with a single senior executive, working closely with the Risk Management function. Who this should be, a new hire or a current senior manager, will depend on a variety of factors.One option to consider would be to assign an existing senior manager this responsibility, whilst a new executive role is designed in the AI Compliance programme’s roadmap. A new hire may be required depending on the scale of the role, the skillsets and knowledge required and possible limitations in the bandwidth of other executives. This will become clear as the AI Compliance programme evolves.
As was the case with data protection in the early days, AI risk management may start off as something of a compliance ‘orphan’, with no single function or individual owning the issue. In the early phases of designing an AI Compliance programme, the Risk Manager will need to engage with a wide group of stakeholders to develop, and then deliver, the AI Compliance programme. In a financial services firm, we would expect this to include, at the very least, representatives from the Risk, Compliance, Legal, IT, HR and Product functions.
A cross-functional team is very strongly advised.
A gap analysis will help the firm determine what existing governance structures, departments, polices, processes, risk categories, metrics, etc., will need to be built or augmented to achieve EUAIA compliance. This will indicate what existing resources are available, or need to be identified, to enable the efficient and effective design and implementation of the AI Compliance programme.
If the assessment is conducted well, it will bring all the different stakeholders to the table to address any alignment issues and highlight any resourcing/skills gaps.
As a minimum, the AI Compliance programme should include:
- Governance and Oversight - oversight of this initiative will require a considerable amount of learning and development for the managers themselves, who may or may not have a background in AI and, in all likelihood, have no background in AI ethics or regulations.
- AI policies, procedures and standards.
- Clearly defined roles and responsibilities across the three lines of defence.
- Early engagement of technology and infrastructure teams. This is especially important if external tooling (IT risk assessment and approvals) or new development work (budget, priorities, resources) will be required as part of the programme.
- Training, education and awareness at all relevant levels in the organisation.
- Compliance monitoring and testing plans.

New Link Consulting and AI & Partners.

New Link Consulting (‘NLC’) has a long history of helping financial services firms manage regulatory risks. We work with firms to develop pragmatic and proportionate risk management frameworks, including governance, policies, procedures, and training.

NLC has partnered with AI & Partners (‘AIP’) to deliver a holistic, front-to-back EUAIA compliance solution. AIP has developed a powerful but intuitive software toolset named “Orthrus”, which automates AI application discovery and classification. The toolset is efficient, secure and robust, standardising processes and workflows across AI compliance initiatives, including all elements from data ingestion, model development, one-click deployment, and model life cycle management, by risk assessing all AI systems.

AI & Partners is a leading professional services firm specialising in assisting companies subject to the EU AI Act. Their expertise lies in providing comprehensive advisory and consultancy services tailored to ensure compliance with regulatory requirements. Additionally, AIP offers a cutting-edge regtech platform designed to identify and risk-classify AI systems, enabling clients to navigate the complex landscape of AI regulation effectively.

For further information please contact Peter Booke at pbrooke@new-linkconsulting.com