Why Google is Considering Reducing Webpage Crawl Rates
Google could potentially decrease the rate of webpage crawling as it becomes more mindful of the sustainability of indexing and crawling.
The topic was addressed by Google’s Search Relations team, which comprises John Mueller, Martin Splitt, and Gary Illyes. They discussed what Google has in store for the future in the latest episode of the Search Off the Record podcast. In this context, “low-hanging fruits” refer to needless web crawling, such as crawling web pages that haven’t been updated recently. One of the topics they cover is crawling and indexing, which has reportedly been reduced over the past year according to SEO professionals and website owners.
In an effort to conserve computing resources and make crawling more sustainable, Google is planning to prioritize this area in the upcoming year. As a result, it is crucial to understand how this change may impact your website’s performance in search results.
How Sustainable is Crawling & Indexing
The sustainability of crawling & indexing has been questioned repeatedly. Since Googlebot crawling and indexing happens virtually, most people would assume it has little to no effect on the environment. This issue was brought to attention when Ilyes stated that computing isn’t sustainable in general. Here’s what he had to say;
“What I mean is that computing, in general, is not really sustainable. And if you think of Bitcoin, for example, Bitcoin mining has real impact on the environment that you can actually measure, especially if the electricity is coming from coal plants or other less sustainable plants.
We are carbon-free, since I don’t even know, 2007 or something, 2009, but it doesn’t mean that we can’t reduce even more our footprint on the environment. And crawling is one of those things that early on, we could chop off some low-hanging fruits.”
How Does Google Plan to Make Crawling More Sustainable?
Ilyes went on to explain that web crawling can be made more sustainable by cutting down on refresh crawls. There are two types of Googlebot crawling: crawling to refresh existing content and crawling to discover new content. Google has considered scaling back on crawling to refresh content.
Illyes continues:
“One thing that we do, and we might not need to do that much, is refresh crawls. Which means that once we discovered a document, a URL, then we go, we crawl it, and then, eventually, we are going to go back and revisit that URL. That is a refresh crawl.
And then every single time we go back to that one URL, that will always be a refresh crawl. Now, how often do we need to go back to that URL?”
He went on to give an example of certain websites that warrant a significant number of refresh crawls for certain parts of websites but not others. The Gold Coast Bulletin for example is constantly providing updates to its homepage with new content. So it is deserving of a large number of refresh crawls. However, it is unlikely that the Gold Coast Bulletin is editing its About Us page to the same frequency as its homepage. So fewer refresh crawls are required for this webpage.
What Could a Reduction In Crawl Rate Mean For Your Website?
If Google were to cut down on refresh crawls, here’s the impact that could have on your website.
There’s a belief out there that a high crawl rate is a positive SEO signal, even if you’re not updating your content as often as Google is crawling it. According to Illyes, this may be a misconception as content will not necessarily be ranked better because it is crawled more.
Because of this change, Right now is the time to ensure that your business’s SEO practices are optimised effectively for these new changes. There is a lot of speculation as to when these changes will come into place, but the majority of people agree that this change may lead to having to update SEO practices so as to not decrease your webpage’s google ranking. The idea for this change is to learn which pages require refresh crawls and which pages do not. This just means the pages you change more often will refresh at a higher rate than web pages that do not require it. Check out the SEO section of our website to enquire about having your web page’s SEO optimized to the highest level by professionals. Book a FREE Consultation
For more information regarding this change, check out this podcast that discusses this entire situation.
