Web crawler

A web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing.

Back to SEO Foundations →

Source: Wikipedia · Image: Wikimedia Commons · CC BY-SA 3.0

GOTSOT

Rank smarter — SEO guides, keyword and content tips, and the tools every marketer needs.

Get new guides by emailOccasional emails. No spam. Unsubscribe anytime.

InformationAbout Contact Privacy Policy Terms of Service Disclaimer Affiliate Disclosure Cookie Policy

Every species, image, fact, and figure is drawn from Wikipedia and Wikimedia Commons, used under public-domain and Creative Commons licenses. No content is AI-generated.