Yes. If you wish to block an individual file in your file manager, such as a PDF document, from being indexed by search engines, you will need to select a connected subdomain for the file(s) and use the file URL to be blocked from crawlers.
The default file URL looks like this. Notice 'CDN' as the subdomain:
Selecting a connected subdomain looks like this:
To select a connected subdomain for your file:
- Navigate in HubSpot Marketing to Content > File Manager
- Locate the file you wish to block from crawlers > click on the file name.
- In the right sidebar, click the File URL dropdown. A connected subdomain will display your company's domain, such as www.your-company.com or info.your-company.org. Depending on the domains connected to HubSpot, you might have multiple options in this dropdown.
- Select the file URL containing your company's connected subdomain > click Copy URL.
- You can now use this URL when blocking the file from being indexed by search engines via a Robots.txt file. To learn how to install a Robots.txt file, click here. Note: If you have shared the file online using more than one connected subdomain included in the dropdown, copy the file URL for each connected subdomain to then add to your Robots.txt file.