COS-General

What is the robots.txt in my Content Settings?

Last updated: May 2, 2016

Available For:

Product: HubSpot Marketing
Subscription: Basic, Professional, & Enterprise

HubSpot’s COS platform makes it easy for you to control the way your HubSpot hosted content is indexed by search engine crawlers with the Robots.txt file editor interface.

Located under Content > Content Settings is the Robots.txt (Advanced) section where you can specify permissions for all pages on your site, some pages on your site, or single pages on your website by adjusting the Disallow rules for crawling agents. Once a rule has been entered Google, Bing, Yahoo!, and any other search engine that follows robots.txt best practices will have a clear understand of what they should and should not index for your domain.

To learn more about how a robots.txt file works, check out this helpful article.