The European Union Agency for Fundamental Rights (FRA) published a report on the bias in algorithms; how it appears and its impacts.  The report is centred around case studies on algorithms in predictive policing and offensive speech detection. However, the issues and potential actions identified are relevant to other use cases for algorithms and AI systems.  Here we highlight some of the key issues with algorithmic bias in those case studies and what the FRA says EU institutions need to do.

Issues with algorithms

The report discusses various risks that lead to algorithmic bias; here we pick out a couple:

Feedback Loops in predictive policing

There is a risk that feedback loops in algorithms and AI systems reinforce and exacerbate bias.

For example:

A feedback loop occurs when predictions made by a system influence the data that are used to update the same system. It means that algorithms influence algorithms, because their recommendations and predictions influence the reality on the ground. This reality then becomes the basis for data collection to update algorithms. Therefore, the output of the system becomes the future input into the very same system.

This is a particular issue for high-risk AI systems, such as predictive policing (using algorithms to predict victims or suspects of crime, or crime hotspots, to inform policing policy).

Several factors can contribute to the formation of feedback loops:

1. data quality, such as low and varying crime reporting rates resulting in underestimates of the 'true crime rate';
2. different rates of detection.  Some crimes are easier to detect, such as car crimes which often lead to a police report in order to make an insurance claim.  In contrast, fraud and financial crime may be harder to detect.  The risk is that certain population groups may be more often associated with crimes that are easier to detect, leading to biased predictions over time; and
3. improper use of machine learning, such as an over-reliance on training data (overfitting).
   

Data and transparency issues in offensive speech detection

The report also looked at online hate speech detection systems which use machine learning and natural language processing (NLP) to identify potential offensive speech. The report identifies several reasons why such detection tools can result in biased results:

• tools may be developed for a particular language and do not reflect different languages (e.g. Italian which uses gendered nouns);
• similarly, datasets for different languages may vary in quality and quantity, affecting how models are trained;
• where AI tools are developed by large companies, assessing levels of algorithmic bias can be more difficult due to a lack of transparency, with limited access to the tools' full documentation, datasets or models; and
• data used in NLP tools can be difficult to share, in part, because 'researchers are overly cautious and avoid sharing data, often because they lack knowledge of data protection rules'.

What needs to be done?

The FRA called for EU institutions and countries to:

• Test for bias – 'The EU legislator should make sure that regular assessments by providers and users are mandatory and part of the risk assessment and management requirements for high-risk algorithms'.
• Provide guidance on sensitive data – the EU AI Act (article 10(5) sets out the grounds for collecting sensitive data strictly necessary 'for the purposes of ensuring bias monitoring, detection and correction in relation to the high-risk AI systems'. The FRA consider that 'additional implementing guidance' should be considered. 
• Assess ethnic and gender biases – EU laws, including the EU AI Act and Digital Services Act should require algorithmic discrimination assessments with 'the requirement for increased transparency and assessments of algorithms being the first step towards safeguarding against discrimination, companies and public bodies using speech detection should be required to share the information necessary to assess bias with relevant oversight bodies and – to the extent possible – publicly.' 
• Consider all grounds of discrimination – Potential discrimination is wide-ranging.  To be effective algorithms need to consider all prohibited grounds of discrimination rather than a handful.
• Promote language diversity – EU should fund and promote NLP research on a range of EU languages in order to promote the use of properly tested, documented and maintained language tools for all official EU languages.  The EU and Members States should also consider building a repository of data for bias testing in NLP.

Why now?

Algorithmic bias is a real and significant concern.  The FRA point to how the Netherlands tax authorities used algorithms that mistakenly labelled around 26,000 parents as having committed fraud in their childcare benefit applications, causing financial and psychological harm. The algorithms were found to be discriminatory.  With the growing use and capabilities of algorithmic and AI systems, in particular those which impact human rights, there are growing concerns to address the risk of algorithmic bias.  The FRA recognise that the EU's proposed AI Act is an opportunity to address those concerns.

If you would like to discuss how current or future regulations impact what you do with AI, please contact Tom Whittaker or Brian Wong.