Where Google bot spends its finite budget for discovery and indexing
Googlebot has a fixed crawl budget per site, determined roughly by your domain authority, freshness expectations, and server response time. High-authority sites get higher budgets. Google allocates this budget across your pages using signals about importance, freshness, and user traffic. It concentrates budget on high-authority pages close to the homepage because those are crawled most frequently by users and competitors. It crawls recently updated pages more frequently than stale pages. It crawls pages with user traffic more frequently than pages with zero traffic.
The consequence is harsh for large sites. A 100,000-page site with a small crawl budget might discover only 10,000 pages in a year and index only 5,000. The other 95,000 pages languish undiscovered or indexed but stale. Google can't see your new content or content changes if your crawl budget is exhausted.
Prioritizing your site's crawlability through hierarchy and signaling
The solution is explicit prioritization. Every page you want indexed quickly should be discoverable in 2-3 clicks from the homepage. Add important pages to your sitemap and submit it to Google Search Console. Make sure your internal linking explicitly points to pages you want to prioritize. If a page is important but buried, use internal links to surface it.
Reduce crawl waste. Delete duplicate pages and use rel=canonical to consolidate them. Block crawling of low-value pages (archives, temporary pages, search filters, session parameters) using robots.txt. A leaner, better-structured site needs less crawl budget to index completely. Set a reasonable crawl rate in Search Console. If your server is slow, Google limits crawl rate to avoid overloading. Improving server response time lets Google crawl more pages with the same budget. Some large sites gain 10-20% more crawl coverage just by improving server speed from 2 seconds to 500 ms average response time.