Some schools in the UK have installed AI-driven software that “actively listens” to students in the bathrooms.
Administrators say they want to use this software to address vaping, bullying, and other misbehavior in the bathrooms. Phrases that can trigger assistance include "help me" and "stop it." One of the companies behind the sensors, Triton, argues that it seeks to “provide an additional layer of security against threats like bullying or sexual assault in these areas, reinforcing a safe school environment … to enhance safety, not monitor everyday conversations.” But digital advocates say that “secretly monitoring school bathrooms is a gross violation of children’s privacy and would make pupils and parents deeply uncomfortable." This tech could be used to discriminate against students based on gender as well.
Related Issues by Justice Area
Amazon's AI recruitment tool discriminated against women.
The online giant created its own recruitment tool to help evaluate the resumes of people who for software developer positions. It was soon discovered that the algorithm was poorly ranking applications from people it assumed were women. This is because it was trained on a dataset that consisted primarily of men’s resumes that were submitted to the company over the past decade. The system was taught to penalized any resume that included the word “women’s” or listed that the candidate graduated from a couple of specific women’s-only universities. Use of the software has been discontinued.
Economic Justice Gender Justice Issue 2019Deepfakes overwhelmingly target people who identify as women.
A 2019 study from cybersecurity company DeepTrace found that 96 of the AI-powered deepfake videos circulating on the internet feature pornographic content. Much of it depicts women in the nude or performing sex acts without their consent. While this technology is frequently used to denigrate celebrities like rapper Megan Thee Stallion or singer Taylor Swift, but it’s also used to manipulate and coerce everyday people.
Gender Justice Issue 2023A 2023 report from Internet Watch Foundation AI is increasingly being used to generate child sexual abuse material (CSAM).
Researchers examined 20,254 AI-generated images that were posted to one dark web CSAM forum over the course of one month and a preliminary review found that 11,108 of them were likely criminal. Ultimately, 2,978 were deemed criminal per UK law. They also concluded that the AI-generated images are increasingly becoming visually indistinguishable from CSAM that involve real people, that much of it focuses on celebrity children and people who have previously been abused, and that there is potential for massive growth in this criminal area.
Community Health and Collective Security Human Rights Issue 2023AI could be used to better meet the needs of the disabled, but there are currently many instances where it actively works against the disabled community.
In 2023, researchers at Pennsylvania State University published “Automated Ableism: An Exploration of Explicit Disability Biases in Artificial Intelligence as a Service (AIaaS) Sentiment and Toxicity Analysis Models,” which explores the bias embedded in several natural language processing (NLP) algorithms and models. They found that every single public model they tested “exhibited significant bias against disability,” classifying sentences as negative and toxic simply because they contained references to disability, ignoring context and the actual lived experiences of disabled people.
Community Health and Collective Security Disability Justice