BetaSync HubSpot data to Amazon S3

Use the Amazon S3 integration to sync HubSpot CRM data to an Amazon S3 bucket that you own. You can choose which objects, associations, and events to sync, configure a recurring sync schedule, and monitor sync status from within HubSpot.

Each sync exports a full snapshot of the selected data in Parquet format to your Amazon S3 bucket. You can then use the synced data with analytics and data processing tools, such as AWS Glue, Amazon Athena, Amazon EMR, Spark, Databricks, or Python notebooks.

Please note: the Amazon S3 integration is available to new Amazon S3 users who are not using the legacy Amazon S3 integration. If you're a Super Admin, you can opt your account into the new Amazon S3 integration beta.

Before you get started

Before you begin syncing data to Amazon S3, review the following requirements and limitations.

Understand requirements

• If you're a Super Admin, opt your account into the Amazon S3 integration beta.
• You must create an AWS Identity and Access Management (IAM) role associated with your HubSpot account.

Understand limitations & considerations

• During beta, data is synced only in Parquet format.
• Each sync creates a complete snapshot of the selected data.
• HubSpot creates a new folder for each sync and doesn't delete previous syncs from your S3 bucket.
• Sensitive Data isn't supported to sync.

Create the IAM role in AWS

To configure access permissions for HubSpot in the AWS Management Console:

1. Log into the AWS Management Console.
2. On the home dashboard, click Services > Security, Identity, & Compliance > IAM.
3. On the left sidebar, click Roles.
4. Click Create role.
5. In the Trusted entity type section, select your AWS account.
6. In the AWS account section, select Another AWS account and enter your own AWS account ID. You'll later change the trust relationship to give access to HubSpot.
7. Select Require external ID and input a placeholder ID like 0000. You'll later change the trust relationship for your IAM role and specify the external ID for your storage integration. An external ID is used to grant access to your AWS resources to a third-party app like HubSpot.

8. Click Next.
9. In the Role details section, enter a Role name and Description to identify the role.
10. Click Create role.
11. On the Summary page, copy the ARN value. You will use this value in the next step.

Connect HubSpot and AWS S3

1. In your HubSpot account, click the Marketplace icon in the top navigation bar, then select HubSpot Marketplace.
2. In the search bar, search for and select Amazon S3.
3. Click Install.
4. Select your AWS IAM role, then click Install.
5. Click Next.
6. Enter the ARN value from the AWS IAM role in the Role ARN field.
7. Click Connect to Amazon S3.

Configure bucket permissions

HubSpot needs the following permissions on an AWS S3 bucket and folder to access files in the folder and sub-folders:

• s3:GetBucketLocation
• s3:GetObject
• s3:GetObjectVersion
• s3:ListBucket
• s3:PutObject
• s3:DeleteObject

After connecting the app, grant HubSpot permission to sync data to your S3 bucket.

1. In your HubSpot account, click the settings icon in the top navigation bar. In the left sidebar menu, navigate to Integrations > Connected Apps.
2. Click Amazon S3.
3. In the right panel, click Select for the Amazon S3 account you want to sync data to.
4. On the Permissions page, click Copy to copy the example permission policy and attach it to the AWS IAM role you created earlier. Please note to update the permission policy with your bucket names.

5. Log into the AWS Management Console.
6. On the home dashboard, click Services > Security, Identity, & Compliance > IAM.
7. On the left sidebar, click Account settings.
8. In the Security Token Service (STS) section, look for the AWS region that corresponds to the region your AWS account is located in. Click to toggle the switch on to activate the region.
9. On the left side panel, click Policies.
10. Click Create policy.
11. On the JSON tab, click Add new statement.
12. Add the policy you copied in Step 4 that will give HubSpot the permissions it needs to load or unload data using a single bucket and folder path.

13. Click Next.
14. In the Review and create section, add the Policy name and an optional description.
15. Click Create policy.

Configure the destination bucket

1. Back on the Permissions page in HubSpot, in the Bucket field, enter the bucket name.
2. Click Test connection.

3. If the connection test is successful, click Next.

Configure the sync

1. On the Data source page, select the data that you want to sync.
2. Select the checkboxes for the objects, associations, and events you want to sync from HubSpot to Amazon S3.

3. Click Next.
4. On the Details page, click the Sync frequency dropdown menu and select the frequency that you want data to sync from HubSpot to Amazon S3.

5. Click Start sync.

Manage the sync

After setting up a sync and data starts syncing from HubSpot to Amazon S3, you can review its status, identify errors, update settings, or archive the sync.

To review and manage the sync:

1. In your HubSpot account, click More, then navigate to Data Management > Data Integration. If More doesn't appear in your account, navigate to Data Management > Data Integration directly.
2. Navigate to the App syncs tab and locate your Amazon S3 sync in the table.
3. Review the sync status:

• • Syncing: a sync run is in progress.
  • Synced: the last sync completed successfully.
  • Failed: the last sync encountered an error.
  • Archived: the sync configuration was removed.

4. Click the sync name to view more details, including successfully synced and failed records.

To edit or remove a sync:

1. In the table, click the Amazon S3 sync.
2. In the top right, click the Actions dropdown menu, then select one of the following:

• • Edit sync: update the sync settings.
  • Archive sync: delete the sync configuration.

Data format for syncing between HubSpot and Amazon S3

Available data for syncing

You can sync the following data from HubSpot to AWS S3:

• Objects: standard and custom objects, such as contacts, companies, tickets, or products.
• Records: individual object records.
• Properties: properties and their values on the object records, including the property history.
• Associations: all association types and associated records.
• Events: selected standard HubSpot events and custom behavioral events, including selected event properties.

Object data of different types are organized and available in two ways:

• Individual: each object type is stored in its own table and has only records of the same object type. For example, objects_contacts contains only contact records.
• Combined: all records for all object types are combined into one database object. Thus, when querying, you need to specify a filter like “WHERE objectTypeId=’0-1’

Custom objects will be assigned an ID at the time of creation, and will be identified as 2-unique_ID. For example, 2-12345.

S3 folder structure

HubSpot writes each successful sync to a HubSpot-managed prefix in your Amazon S3 bucket using the following structure:

s3://<bucket-name>/hubspot-dataout/<account-id>/<run-id>/<table-name>/<file-name>

For example:

s3://my-company-hubspot-data/hubspot-dataout/12345678/98765/OBJECT_WITH_OBJECT_PROPERTIES/part-000.parquet

s3://my-company-hubspot-data/hubspot-dataout/12345678/98765/EVENTS_VISITED_PAGE/part-000.parquet

s3://my-company-hubspot-data/hubspot-dataout/12345678/98765/ASSOCIATIONS_CONTACTS_TO_COMPANIES/part-000.parquet

Each sync creates a new run-id folder. HubSpot doesn't delete data from previous sync runs.

If you want to automatically remove older sync, configure an Amazon S3 lifecycle policy for the bucket or prefix.

HubSpot generates file names for each sync run, and file names may differ between sync runs. Don't configure downstream processes to depend on specific file names. Instead, read all files in the appropriate table-name folder for the export run that you want to process.

Object files

If you select object data, HubSpot exports the following folders.

OBJECTS_PROPERTIES

One row per object property value.

OBJECT_WITH_OBJECT_PROPERTIES

One row per object record, with selected property values grouped into a single PROPERTIES column.

To read a specific object type from this folder, filter by OBJECTTYPEID.

Association files

If you select association data, HubSpot syncs association records and association definitions.

ASSOCIATIONS

One row per selected association between two object records.

ASSOCIATION_DEFINITIONS

One row per selected association definition.

ASSOCIATIONS_<FROM_OBJECT_TYPE>_TO_<TO_OBJECT_TYPE>

For selected association pairs, HubSpot also syncs folders for specific association pairs, such as ASSOCIATIONS_CONTACTS_TO_COMPANIES. These folders contain association records and the associated definition metadata.

Event files

If you select event data, HubSpot exports one folder for each event type. Folder names use the format EVENTS_<EVENT_NAME>, such as EVENTS_VISITED_PAGE.

Each event folder contains standard event columns and a column for each event property selected during setup.