Whenever a sync runs, Hightouch tracks all records that the destination rejects using their primary key. Hightouch then automatically retries syncing these records two ways:
- Within an Active Sync: If Hightouch receives any record-level errors during an active sync, it automatically retries those records at an exponential backoff until it meets a destination-dependent failure threshold. Hightouch's retry process is similar to the one described by AWS. Some destinations support split retries as part of the active sync retry strategy.
- On Subsequent Syncs: Hightouch keeps an index of all failed records and automatically retries them on every subsequent sync until they succeed.
Example retry scenario
Imagine you are trying to sync 10,000 records to Salesforce in a single sync, and five of the records are locked within Salesforce at the beginning of the sync.
- During that sync run, imagine two of those five records become available for update. Because Hightouch performs retries during an active sync, these two will update successfully during the sync run.
- On the subsequent sync run, Hightouch automatically retries the remaining three records. Assuming they're unlocked by that point, they will successfully update on the subsequent sync. Hightouch continues to retry the failed records until they are successfully synced.
Split retries
Some destinations count all records in a batch request as rejected if the request contains a single invalid record. For some of these, Hightouch provides you the option of enabling split retries.
If you enable split retries, Hightouch splits the rejected batch in half and retries each sub-batch separately. If either of those sub-batches fail, Hightouch splits it in half and retries each sub-batch separately. Hightouch repeats this process until the destination has marked each record in the batch as successful or rejected.
The split retry strategy allows you to pinpoint which records are getting rejected with which errors. It also reduces the number of valid records that get retried. Split retries are part of the active sync retry strategy. Hightouch retries records that are still rejected at the end of the split retry process in the subsequent sync.
To see if a destination supports split retries, refer to its docs page.
Enabling split retries may lead to longer sync times.
FAQ
If part of my sync failed, does Hightouch retry every record in the sync?
Hightouch generally only retries failed records. For example, if you try to sync 1,000 records to Salesforce, but only 600 pass, Hightouch retries only the 400 that failed.
The exception to this is when a destination marks an entire batch as failed, even if only some of the records are invalid. In that case, Hightouch retries the entire failed batch. Enabling split retries can help to reduce the number of valid records that get retried.
How can I be alerted to failing records or syncs?
Hightouch retries failed records until the underlying issue is addressed. To become aware of failed records or syncs and fix them, it's best to set up alerting.
How can I avoid Hightouch retrying invalid rows?
If one or more of your records have malformed data that won't be accepted by the destination, Hightouch retries them and they will continue to be rejected.
If you don't want to sync these records to the destination, you can edit your model's definition to filter out these rows. The SQL query you use to filter out invalid rows will depend on your destination's definition of accepted values.
Why is Hightouch retrying a removed row?
If rows are removed from your model query result and you've configured delete behavior for your sync, Hightouch sends requests to your destination to clear or delete removed records. Hightouch either clears or deletes the removed records from the destination based on the delete behavior setting you configured.
If a sync is unable to clear or delete certain destination records, Hightouch retries the request during the following sync run. This can happen because the destination rejects the request or if the destination's API is inaccessible.
If you don't want the sync to retry removed rows in future sync runs, you can choose one of the following options:
- Set delete behavior to Do nothing. Keep in mind that this affects all rows removed from the model's query results, not just the ones being retried.
- Temporarily set delete behavior to Do nothing, run the sync manually once, and then reconfigure the sync to the delete behavior setting you would like. This clears out the removed rows from the diff file created by CDC.
- Reset change data capture (CDC) without a backfill.
Please keep in mind that:
- with option 2), during the run where delete behavior is set to Do nothing, no model rows that CDC marks as "removed" are synced to the destination
- with option 3), any changes to your model (added, changed, or removed rows) between the previous run and the Reset CDC run aren't synced to the destination
To minimize the risk of model data not being synced to the destination, we recommend triggering a regular sync run right before following through with options 2) or 3).