What is a web crawler, really?

In this episode of Search Off the Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way.    Resources: Episode transcript → https://goo.gle/sotr070-transcript  Managing your crawl budget → https://goo.gle/3IzRZxl Dave Smart on LinkedIn → https://goo.gle/3wPSuRA Tame the Bots → https://goo.gle/4cfCQ1P Search Central Help Forum → https://goo.gle/sc-forum Indexing API docs → https://goo.gle/3v8yVU0   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.   #SOTRpodcast   Speaker: Gary Illyes, Lizzi Sassman Products Mentioned: Search Off The Record

Om Podcasten

Search Off the Record takes you behind the scenes of Google Search and its inner workings! In each episode, the folks from the Search Relations team will give you background info on the decision-making behind launches, feature prioritization in Search Console, and the projects Google Search teams are working on. They will share fun stories from the many conferences they attend as well as from their day-to-day working life at Google. They will also dive into the currently trending conversations in the SEO community at large. Have a listen!