The Shared AI-Training Signal Is Coming. Keep Your Current Crawler Rules.

2026-10-31

Maestro’s take. A shared way to express AI-training preferences would reduce crawler-by-crawler maintenance. It does not yet replace the controls your site relies on. Keep those rules until each provider documents its rollout.

TL;DR

On October 30, OpenAI committed to honoring a common robots.txt signal, with implementation planned in the coming months. Existing OpenAI crawler controls remain in place.

Google announced planned support in 2027. It says an opt-out using the shared token would be interpreted like disallowing Google-Extended when Google-Extended is not explicitly specified. Provider-specific choices would still be available.

The proposed token is __usage_ai-training. The IETF tracker identifies Kevin Kelley’s document as an individual Internet-Draft, dated October 30. It is not an endorsed IETF standard.

Why Maestro users care

Teams building websites need to distinguish three decisions: whether a crawler may fetch a page, whether retrieved content may be used for generative-model training, and which providers have implemented the preference.

The draft’s sections 4.3 and 4.4 keep training preferences separate from crawler rules. The token itself is not a crawling directive. Publishing it does not install an access barrier.

Maestro sees a useful reduction in maintenance if adoption follows. The immediate risk is simpler: someone replaces working provider rules with a shared signal before that provider supports it. A cleaner file would then express less than its owner intended.

Keep private material behind authentication. Use robots.txt to communicate crawler and usage preferences, not to protect secrets.

One thing to watch

Inventory your current provider-specific rules and the decision behind each one. Track the draft revision and each provider’s implementation notice. When support arrives, check precedence before replacing anything—especially where your choices differ between providers.

Published October 31, 2026 · Brief and Maestro commentary · Tags: workflow, web, models

Sources considered: OpenAI’s rollout commitment · Google’s proposed behavior and timing · Draft status · Version 00 semantics.