AI hiring algorithms come complete with dangerous bias.
About 70 percent of companies (and 99 percent of Fortune 500 companies) around the world use AI-powered software to make hiring decisions and track employee productivity. The problem? The tools work by identifying and replicating patterns around who was previously hired, which means they perpetuate the bias embedded in the system, locking marginalized populations out of employment. This is particularly tough for disabled people, people of color, and disabled people of color, who are often subject to employment discrimination.
Related Issues by Justice Area
AI could be used to better meet the needs of the disabled, but there are currently many instances where it actively works against the disabled community.
In 2023, researchers at Pennsylvania State University published “Automated Ableism: An Exploration of Explicit Disability Biases in Artificial Intelligence as a Service (AIaaS) Sentiment and Toxicity Analysis Models,” which explores the bias embedded in several natural language processing (NLP) algorithms and models. They found that every single public model they tested “exhibited significant bias against disability,” classifying sentences as negative and toxic simply because they contained references to disability, ignoring context and the actual lived experiences of disabled people.
Community Health and Collective Security Disability Justice Issue 2017AI systems reflect the culture's bias against the disabled.
The Allegheny County Department of Human Services in the state of Pennsylvania in the United States uses an AI system that residents allege incorrectly flags disabled parents as being neglectful, removing their children from their homes with no actual evidence of neglect. It is currently under investigation by the United States Department of Justice.
Community Health and Collective Security Disability Justice Human Rights Issue 2023Medicare Advantage insurance plans use AI to determine what care it will cover for its 31 million elderly subscribers in the United States.
Journalists at Stat found that companies are specifically using AI systems to deny coverage for care. The massive problem: the algorithms are a black box that can’t be peered into, making it nearly impossible for patients to fight for health care coverage when they don’t know why they were denied it in the first place.
Community Health and Collective Security Disability Justice Economic Justice Issue 2020In June 2020, the north Indian state of Haryana began using a new algorithmic system, called the Family Identity Data Repository or the Parivar Pehchan Patra (PPP) database, to determine eligibility for people requesting financial assistance because they are unable to earn enough to survive for various reasons.
It works by pulling demographic and socioeconomic information for families via several linked government databases, using data on birth, death, marriage, employment, property ownership, income tax payments and more. But it is incorrectly marking residents as dead and blocking them from much-needed funding. Meanwhile, human workers don't have the power to override bad decisions made by the system. A 2023 government study on the system found that "it stopped the pensions of 277,115 elderly citizens and 52,479 widows in a span of three years because they were 'dead.' However, several thousands of these beneficiaries were actually alive and had been wrongfully declared dead either due to incorrect data fed into the PPP database or wrong predictions made by the algorithm.”
Disability Justice Economic Justice