If an SEO crawler can't index a page, it will return a crawling error. This can happen with the crawlers in HubSpot's SEO and import tools, as well as external crawlers like Semrush. The steps for resolving a crawling error depend on the error and where the page is hosted.
HubSpot's SEO tools crawling a HubSpot page
You can view SEO recommendations on the Optimization tab of a page or post's performance details. If there are issues crawling the page, you may see one of the following error messages:
Status 301: Moved Permanently - a 301 redirect is preventing the crawler from accessing the content.
Status 302: Object moved - a 302 (temporary) redirect is preventing the crawler from accessing the content.
Status 403: Forbidden - the server can be reached, but access to content is denied.
Status 404: Not Found - the crawler is unable to find a live version of the content because it was deleted or moved.
Crawl of [site] blocked by robots.txt - a robots.txt file is blocking the content from being indexed.
Scan blocked by robots.txt file: if your external page is excluded from indexing by your robots.txt file, add our crawler’s user agent “HubSpot Crawler” as an exemption.Learn more about working with a robots.txt file here.
Robots.txt file couldn't be retrieved: if HubSpot's crawlers can't access your site's robots.txt file, verify that the robots.txt file is accessible and in the top-level directory of your site. Learn more about working with a robots.txt file here.
The crawler isn't able to scan this URL: if HubSpot's crawlers can't crawl a specific URL, try the following troubleshooting steps:
Verify that the URl has been entered correctly.
Verify that the page being crawled is currently live.