What Crawl Budget Means for Google search bot
Crawl rate limit:
The crawl rate can go up and down based on a couple of factors:
- UP: If the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl.
- Down: If the site slows down or responds with server errors, the limit goes down and Googlebot crawls less.
Crawl demand:
The two factors that play a significant role in determining crawl demand are:
- Popularity: URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in Google index.
- Staleness: our systems attempt to prevent URLs from becoming stale in the index.
Factors affecting crawl budget:
According to Google analysis, having many low-value-add URLs can negatively affect a site's crawling and indexing.
- Faceted navigation and session identifiers
- On-site duplicate content
- Soft error pages
- Hacked pages
- Infinite spaces and proxies
- Low quality and spam content
Source: https://developers.google.com/search/blog/2017/01/what-crawl-budget-means-for-googlebot
Comments
Post a Comment