Google is a technology company best known for its search services and tools. Google Search uses Googlebot, the general name for its web crawlers, to discover and access website content for indexing. The crawlers include Googlebot Smartphone, which simulates a mobile device, and Googlebot Desktop, which simulates a desktop computer. Site owners can identify the crawler type through its HTTP user-agent request header, although both types follow the same Googlebot product token in robots.txt and cannot be targeted separately through that file.

For most websites, Google primarily indexes mobile content, so most Googlebot requests come from the smartphone crawler. Crawling generally occurs at a measured pace, but administrators can reduce the crawl rate if their servers struggle to handle requests. Googlebot processes the first 2 MB of supported file types and the first 64 MB of a PDF; referenced resources such as CSS and JavaScript are fetched separately under the applicable size limits.