Search patent of the week: Crawl algorithm
In this episode, a Google patent is discussed. The patent describes a new patent from Google for a crawl algorithm that aims to make web crawling more efficient. The algorithm takes into account the available bandwidth and determines a crawl value for each web page to decide when it should be updated in the cache. The patent may use machine learning and takes into account change signals from web pages to optimize crawl events and conserve resources. The system can divide web pages into shards and allocate a portion of the bandwidth to each shard.https://www.kopp-online-marketing.com/patents-papers/crawl-algorithm