A question that often pops up for sites that are chasing “hot topics” is: how to get Google to crawl our site faster? This could be either because you have a big site and want to see changes faster or because you’re optimizing a news site. There are a few things you can do to improve your crawl speed. We’ll cover them in this post.
There are a couple of possible reasons why Google is slow when spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The other reasons are technical: it has too much to crawl on your site, your site is too slow, or it’s encountering too many errors.
Your site doesn’t have enough authority
When your site doesn’t have enough quality inbound links, Google will not crawl your site very quickly. You can imagine it doesn’t necessarily want to spend much time on a site that it doesn’t deem important. When your site is brand new, this will be the case.
So, when you need more crawl action on your site, start doing some link building. We’ve written about that topic before, I’d suggest reading these three posts as I won’t go into it any further here:
Link building from a holistic SEO perspective
Link building, what not to do
6 steps to a successful link building strategy
If the above three posts are not enough for you, there’s more in our link building archive .
Technical reasons for slow crawling
The technical reasons for Google to crawl your site slowly can be divided into three groups: your site is too slow, you have too many errors, or you have too many URLs.
Your server is slow
The main reason why we see Google crawling sites slowly, is when the site itself is actually slow. Google actually indicates this to you when this is the case, on the “crawl rate” page of Google Search Console. If this is the case, Google might show you a “faster” option:
Our advice: set it to “Faster”, but don’t stop with that: make your server respond faster, either by upgrading your hosting, or improving your site’s caching. Chances are that when Google is suffering long load times, your users are as well.Optimize your site for Google News! Create XML News Sitemaps, editors’ picks RSS feeds and more with Yoast News SEO! »
Buy now » Info
Too many errors on your site
If you have a lot of errors on your site for Google, Google will start crawling slowly too. To speed up the crawl process, fix those errors. Simply 301 redirect those erroring pages to proper URLs on your site.
If you don’t know where to find those errors: use the Google Search Console integration in Yoast SEO. Or log into Google Search Console. If you have access to your site’s access logs, you can also look at those, preferably with a tool like Screaming Frog’s Log file analyzer .
To prevent your site from being crawled slowly, it’s important that you regularly look at your site’s errors and fix them. We have a more extensive article on fixing 404 errors .
When your site is down too much, Google will slow down incredibly hard. If this happens because of your hosting, switch hosting. Make sure to check on your site’s uptime with a tool like Pingdom.
Too many URLs
If you simply have too many URLs on your site, Google might crawl a lot but it will never be enough. This can happen because of faceted search navigation for instance, or another system on your site that simply generates too many URLs. To figure out whether this is the case for you, it’s always wise to regularly crawl your own site. You can either do that manually with Screaming Frog’s SEO spider , or with a tool like OnPage.org .
Tips to increase your crawl speed
A couple of simple tips to increase your site’s crawl speed:
Using the methods above, find and fix all the errors.
Make sure your site is fast.
Add an XML sitemap to your site and submit it to the search engines.
If all of that fails to improve your crawl speed, start link building!
Specifically for news sites, if you want to be indexed faster, you should also look at our News SEO plugin . When you’re in (or want to be in) Google News, it can help make sure you meet all the technical requirements.
Read more: ‘Site speed: tools and suggestions’ »
Xero Media Services 