Addressing Bias in Machine Learning Based Decision Making
Industries like retail, advertisement, healthcare, finance, and even government provided services are being transformed by the application of Machine Learning (ML) techniques which can often find hidden patterns in data not always apparent to human experts. Although ML based tools can be used in many situations, in the human context, their inferences often have serious consequences for people. For example, there are instances when loan approvals or lengths of prison sentences, determined by ML algorithms, have been shown to be biased against certain population groups.
Biases can be introduced in an ML system in several ways:
Sample bias—when the data sets used for training the system do not contain adequate or correct representation of a target population; for example, when data used for training a model is not representative of real-life variations.
Labeling bias—when the data sets may contain errors in labeling or use an incorrect proxy for an outcome; for example, using zip code as proxies for race, or height and weight as proxies for gender.
Measurement bias—when the datasets may contain inaccuracies in certain data points for a certain population.
Pipeline bias—when biases are introduced by the steps within an ML pipeline such as ingestion, feature engineering, picking one model over another.
Detection and correction of such biases are often complicated by the fact that definitions of fairness are often context dependent. For example, for a mortgage company, trying to optimize for lower delinquency rates may disproportionately affect certain groups. Reasonable arguments can be made for each context, but not all of them can be implemented at the same time.
It is also not often clear how such algorithms are harming people. Research into algorithmic harm of ML based algorithms is still fairly new. However, there are several recommendations and best practices organizations can adopt now to address bias in ML-based development.
Conduct a bias impact assessment
Although there is no universally accepted definition of fairness in a model, it is important to acknowledge the possibility of unethical and unfair decisions resulting from a model and identify possible causes. A framework for such a bias impact assessment consists of documenting the following attributes for ML-based decisions:
Identify the possible negative or unintended outcomes of an automated decision-making system.
Identify the organizational incentives to address algorithmic bias which may include balancing the benefits with legal and reputational risks stemming from biased decisions.
Engage users and stakeholders, including legal and risk professionals and civil society organizations to help understand the impacts of biased decision making.
Know your data
Organizations should understand where the data is coming from and guard against the possible introduction of bias in the data sets. Data harvested from publicly available sources on the internet often have comparatively less representation of minority viewpoints.
Regularly audit for bias
Regular audits, reviewing both input data, and output decisions, can provide insights into an algorithm’s behavior and detect bias early. For high-impact situations it may be often worthwhile to bring in independent evaluators experienced in the specific contexts.
Monitor and retrain the models
Data in the real-world changes continuously. What may have been relatively free of bias at a certain point in time may not continue to be so, as business situations evolve. Organizations should build capability to continuously monitor their models for bias with respect to recent and diverse data sets and retrain the models when necessary.
Aim for transparency
Organizations often choose to buy rather than build domain-specific ML decision-making tools from software providers. These algorithms used in these tools and the data sets used to train the models are often proprietary information and not open to interrogation about their decisions. Organizations must insist that vendors of these tools make it easy to understand how certain decisions are being made and have their models retrained if they are leading to biased decisions.
Prioritize explainability
Explainability is the ability of having an ML-based decision explained to the users of the models in simple language. For example, lenders are required to explain the rationale for their loan approval decisions to their applicants. Building an audit trail for ML-based decisions can protect organizations against charges of discriminations. Organizations should invest in tools and techniques to improve explainability of their ML-based decisions.
Incorporate human oversight
Even after incorporating all the best practices, ML models will continue to come up against real-life scenarios that have not been considered as part of their training. ML models should always be complemented by human moderators for improved fairness in decision making.
As ML-based decisions that impact individuals are becoming increasingly pervasive, regulators worldwide are creating frameworks to force organizations to think about bias and fairness. To comply with these frameworks, companies must build legal and technical safeguards to ensure that the ML-based decision making is applied in socially and ethically responsible ways. The recommendations listed above will help organizations incorporate ethics and fairness considerations along their ML journey.