To configure a Robots.txt file for pages on the HubSpot COS, please follow the instructions below.
- Navigate to Content > Content Settings. Select the Page Templates tab.
- Scroll to the Robots.txt section. Once configured, click Save.
To block specific pages from being crawled, add the 'disallow' page(s) in the Robots.txt field. Only add everything in the URL after the .com portion. When finished, click Save.
To check if this was implemented correctly, navigate to your robots.txt file by going to http://your-subdomain.com/robots.txt. ("your-subdomain.com" is meant to indicate where you place your own subdomain within this link. For example, to check the robots.txt for the subdomain "blog.hubspot.com" you'd want to navigate to http://blog.hubspot.com/robots.txt)