Back to AI Intersections Database

There are no good mechanisms to reliably block AI crawlers while still sharing openly online.


Justice Area(s)

Community Health and Collective Security

Economic Justice

AI Impact(s)

Consumer Rights Violation

Economic Harm and Inequity

Privacy Violation






The core issue is that the current mechanism to block crawlers via robot.txt was designed with search engines in mind and only works per domain, not per individual user or post of a user. As The Verge reports, "For decades, the main focus of robots.txt was on search engines; you’d let them scrape your site and in exchange they’d promise to send people back to you. Now AI has changed the equation: companies around the web are using your site and its data to build massive sets of training data, in order to build models and products that may not acknowledge your existence at all."

Share Page

Related Issues by Justice Area

Issue 2023

AI could be used to better meet the needs of the disabled, but there are currently many instances where it actively works against the disabled community.

In 2023, researchers at Pennsylvania State University published “Automated Ableism: An Exploration of Explicit Disability Biases in Artificial Intelligence as a Service (AIaaS) Sentiment and Toxicity Analysis Models,” which explores the bias embedded in several natural language processing (NLP) algorithms and models. They found that every single public model they tested “exhibited significant bias against disability,” classifying sentences as negative and toxic simply because they contained references to disability, ignoring context and the actual lived experiences of disabled people.

Community Health and Collective Security Disability Justice
Issue 2017

AI systems reflect the culture's bias against the disabled.

The Allegheny County Department of Human Services in the state of Pennsylvania in the United States uses an AI system that residents allege incorrectly flags disabled parents as being neglectful, removing their children from their homes with no actual evidence of neglect. It is currently under investigation by the United States Department of Justice.

Community Health and Collective Security Disability Justice Human Rights
Issue 2023

Medicare Advantage insurance plans use AI to determine what care it will cover for its 31 million elderly subscribers in the United States.

Journalists at Stat found that companies are specifically using AI systems to deny coverage for care. The massive problem: the algorithms are a black box that can’t be peered into, making it nearly impossible for patients to fight for health care coverage when they don’t know why they were denied it in the first place.

Community Health and Collective Security Disability Justice Economic Justice
Issue 2024

AI hiring algorithms come complete with dangerous bias.

About 70 percent of companies (and 99 percent of Fortune 500 companies) around the world use AI-powered software to make hiring decisions and track employee productivity. The problem? The tools work by identifying and replicating patterns around who was previously hired, which means they perpetuate the bias embedded in the system, locking marginalized populations out of employment. This is particularly tough for disabled people, people of color, and disabled people of color, who are often subject to employment discrimination.

Disability Justice Economic Justice

Want to suggest an update to the AI Intersections Database?

Help us make this database a robust resource for movement building! We welcome additions, corrections, and updates to actors, issues, AI impacts, justice areas, contact info, and more.

Contribute to the database

Sign Up for News and Updates

Join our Insights email list to stay up-to-date in the fight for a better internet for all.

AI Intersections Database

This website supports Web Monetization

Read more about our implementation

Mozilla is a global non-profit dedicated to putting you in control of your online experience and shaping the future of the web for the public good. Visit us at Most content available under a Creative Commons license.