From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI

We present a large-scale measurement of emerging Artificial Intelligence (AI)-oriented web permission and descriptor mechanisms across approximately 4M domains. Beyond traditional robots.txt files, we analyze the adoption, structure, and interactions of newer files, including the permission-oriented ai.txt and the descriptor-oriented llms.txt and llms-full.txt. While robots.txt remains the dominant mechanism for signaling crawler permissions, adoption of AI-specific permission files is emerging, especially among technology-focused domains. Directive usage shows a strong emphasis on restricting AI agents, with prevalent disallow rules and relatively few AI-specific permission controls. Crossfile comparisons reveal that conflicts between robots.txt and ai.txt are infrequent, while some llms.txt link targets point to content blocked by access-control files.