COS-General

How do I install a robots.txt file?

Last updated: January 11, 2016

To configure a Robots.txt file for pages on the HubSpot COS, please follow the instructions below.

  • Navigate to Content > Content Settings > Page Publishing
  • Scroll down until you see the section titled Robots.txt (Advanced). Once configured, click Save Changes

Blocking specific pages

For specific pages on the HubSpot COS that you wish to not have crawled, please follow the instructions below.

  • Navigate to Content > Content Settings > Page Publishing
  • Locate the Select a Domain to Edit section > choose a connected subdomain that is used for the page(s) you wish to not crawl
User-added image
  • After selecting your connected subdomain, scroll down to Robots.txt (Advanced) section
  • Add the 'disallow' page(s). Only add everything in the URL after the .com portion. When finished, click on Save changes.

To check if this was implemented correctly, navigate to your robots.txt file by going to http://your-subdomain.com/robots.txt. ("your-subdomain.com" is meant to indicate where you place your own subdomain within this link. For example, to check the robots.txt for the subdomain "blog.hubspot.com" you'd want to navigate to http://blog.hubspot.com/robots.txt)