Reports

Why did I get a "crawl blocked by robots.txt" error in the pages report?

Last updated: November 20, 2017

Applies to:

Marketing Hub Basic, Professional, Enterprise
When attempting to check a page for SEO errors or refresh page data, the following error may appear:
Page Performance Error Message Robots.txt

If the URL contains hs-sites.com, the page is still on the HubSpot staging domain, which is automatically prevented from being indexed by search engines in the robots.txt file. In order to build up the SEO for your website and pages, your content should be on a domain or subdomain that is part of your website. You will be able to see the total page views for pages on the HubSpot staging domain and the URLs are public to share; however, they are blocked from being crawled by search engines.

Another reason this error may appear is if the page is blocking our two crawlers. HubSpot uses two different crawlers for pages and links. The user agent for each of the crawlers is:

  • Pages: "HubSpot Crawler 1.0 http://www.hubspot.com/"
  • Links: "HubSpot Links Crawler 1.0 http://www.hubspot.com/"

To allow the HubSpot crawlers to crawl your page, you may need to modify the robots.txt file you have implemented or you may need to modify how you are blocking crawlers at the server-level.

Was this article helpful?

If you still need help you can get answers from the , or to contact support.