Reports

Why is HubSpot crawling my website?

Last updated: November 21, 2016

Available For:

Marketing: Free, Starter, Basic, Pro, Enterprise
Sales: N/A

HubSpot uses web crawlers to provide you with up-to-date, real-time information in the reporting tools (Page Performance, Sources, Keywords) and to display live previews of your pages. If you’re seeing a spike in visits from HubSpot crawlers in your Google Analytics data, you have a few options to get a clearer sense of your traffic: 

  1. Set up a robots.txt file to stop crawlers from accessing parts of your sites. All of our crawlers are fully compliant with rules you set for your own domains. Setting up a robots.txt file will prevent our crawlers (and others, including Google's) from crawling your pages. You can learn more about setting up a robots.txt file here.

  2. Filter out the specific crawler user-agent in Google Analytics. HubSpot Sources and other HubSpot reporting tools will automatically exclude our crawler traffic, but if you'd like to get a more accurate report of traffic in Google Analytics, follow the instructions to exclude the crawler name as a referrer here.

  3. Filter out all bot traffic from Google Analytics. If you're not certain what the exact name of the crawler that is causing increased traffic to your site, Google provides a checkbox for you to filter out all bot/crawler traffic in general. Read more here.

To learn more some of the HubSpot tools that benefit from our crawler, check out these resources: