Robots.txt

The Robots Exclusion Protocol is a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.

Back to On-Page & Technical →

Source: Wikipedia · Image: Wikimedia Commons · CC0

GOTSOT

Rank smarter — SEO guides, keyword and content tips, and the tools every marketer needs.

Get new guides by emailOccasional emails. No spam. Unsubscribe anytime.

InformationAbout Contact Privacy Policy Terms of Service Disclaimer Affiliate Disclosure Cookie Policy

Every species, image, fact, and figure is drawn from Wikipedia and Wikimedia Commons, used under public-domain and Creative Commons licenses. No content is AI-generated.