Why S3?
Hook has several native integrations for easy and direct access to your organisation's data. In some cases this may not be possible:
Hook does not have a native integration for the system your organisation uses to store product metrics. Find the list of integrations Hook supports here.
Data must be transformed in a different format compared to how it is originally stored, and therefore should be sent separately to Hook through S3.
A bulk data transfer is required to provide history for training Hook's engagement model. This may occur if there's a change in systems that store important subscriptions or metrics data, e.g. switching between two CRM providers.
Sending data through S3 enables Hook to access valuable data that would otherwise be difficult to access.
Data Format Required by Hook
Data sent to Hook through S3 must meet the following requirements:
File format
CSV or JSONL
Frequency of export
Weekly or daily export i.e. time series with data for each week or each day.
Metric Types (if applicable)
Cumulative metrics (can be added up over time) or stock metrics (snapshot metrics for a specific point in time). Ideally not year-to-date metrics because these reset each year so history for these values is lost.
Examples:
Cumulative metrics: total orders, total downloads, total logins.
Stock metrics: active users (this week), seats occupied (on this day).
Mapping to Account IDs
It must be possible to link each row to an account in your organisation's CRM (e.g. Salesforce account ID or HubSpot company ID). We recommend including a Account ID column that specifies the CRM account ID. Alternatively, each row can be linked to a user ID, tenant ID, opportunity/deal ID etc which could then be linked to an account in your CRM.
Timestamp of data
A timestamp for when the data is relevant for. This should be contained in the filename and as a column in the file.
Any date fields included in exports should be in ISO date format (YYYY-MM-DD)
Example:
The file is named
hook_export_20240301represents data from the 1st March 2024.There is a column in the file called
metric_datewhich contains the value 2024-03-01, to show that this data is from the 1st March 2024.
Column names and order remain the same in each export
Column names, and the order of the columns, should be the same for every export (regardless of date). A renamed or reordered column reads as a different field.
Example:
For metrics dated 1st March 2024, the total number of journeys should have the column name Total Journeys. The column name should NOT be Total Journeys as of Mar-01-2024 (or similar).
Number of columns remain the same in each export
This is required so that Hook can reliably ingest the data. If new columns are added that we do not expect this will cause data import issues.
Example:
The file is named hook_export_20240301 (1st March 2024) contains 10 columns. All of these columns, with the same column names, are included in the following file named hook_export_20240302 (2nd March 2024).
1-2 years of historical data (Customers with engagement models only):
This is needed for subscription and metrics data used to train Hook's engagement model, so that it can learn patterns from past data.
View an example data extract that meets the above requirements here.
Keeping your exports consistent
The most important thing for a reliable S3 import is that files for a given object look the same every time. These rules prevent the most common issues:
Always include a header row, and keep it identical. Every file starts with a row of column names. Use the same names, in the same order, in every file for that object. A renamed or reordered column reads as a different field.
Tell us before you change the schema. If you need to add, rename, remove or reorder a column, let your Technical Implementation Consultant know before the first changed file lands. A newly added column will not be mapped correctly until we re-scan the object, and renames or removals are the most common cause of a failed import.
Never send empty files. Do not upload a 0-byte or header-only file. If there is no data for a period, send nothing rather than an empty file. An empty file never deletes data you have already sent.
One object per folder, one format per folder. Keep each object in its own folder (see the folder structure in the steps below) and use a single file format per object. Everything in a folder is read together into one table, so mixing objects or formats produces the wrong shape or a parse error.
Write new files, do not overwrite old ones. Each export should be a new, uniquely-named file (the date-stamped structure below does this). If you overwrite a file that has already been imported, Hook re-reads the whole file and its rows load again as duplicates. Treat past files as final.
Removing records. Hook cannot tell a record has been removed just because it stops appearing in later files, those rows remain. If a record should be retired, include a
deleted_date(or a removed flag) column plus a stable unique ID, so we can drop it.Include a unique ID column in every file, present and populated, so records can be matched and updated rather than duplicated. Let us know which column it is.
How to Send Data to Hook using S3
Format the data following the data requirements described here.
Share a sample extract of this data with your Technical Implementation Consultant. This is an extra check to verify that the data is in the correct format.
Hook will share the following credentials for the S3 bucket securely using LastPass. Remember to keep these credentials secure:
Bucket Name: e.g.
hook-customer-data-production-companyname-imports
Region:eu-west-1Access Key ID:
AKIA...Secret Access Key:
wJalr...Set up automated uploads for these files. Find out more here. Ensure these files are uploaded daily or weekly.
For sending files via s3 (excluding test data) we'd recommend this folder structure:
{object}/{date}/file_name.{file_format}
Give each object (table) its own folder, and add a new dated file to it each export. Everything inside a folder is read together into one table in Hook, so one folder = one object.
e.g.
accounts/2026-01-01/accounts.csv
accounts/2026-01-02/accounts.csv
metrics/2026-01-01/metrics.csv
metrics/2026-01-02/metrics.csv
subscriptions/2026-01-01/subscriptions.csv
Keep accounts in the accounts folder, metrics in the metrics folder, and so on, do not mix different objects in one folder. Only the date changes each export; the object folder name and file name stay the same.
You can send either a full snapshot (your complete current state each time) or only the rows that have changed, whichever suits your export. Please tell us which, so we configure the import to match.
Let the Hook team know once this has been completed.
Speak to your Technical Implementation Consultant for further guidance.
Troubleshooting
Access Denied Error:
Verify you're using the correct credentials
Check you're uploading to the correct bucket name
Confirm you're using the correct profile with the
-profileflag (if using AWS CLI)
Credentials Not Found Error:
Run
aws configure --profileagainCheck credentials are saved in the correct location
File Not Appearing:
Verify upload completed without errors
Check you're looking in the correct bucket/path
Try listing files e.g.
aws s3 ls s3://hook-customer-data/
Example: Data Extract (CSV)
Download the example data extract to see the format required.