We designed the HTTP Request destination to be as flexible as possible. We want our customers to build their own integrations with third-party APIs, internal tools, and other services not yet supported in our growing catalog of 200+ native destinations.
This destination empowers you to integrate Hightouch with almost any API without writing code. Using our visual interface, you can exercise granular control over HTTP endpoints, request payloads, rate limits, concurrency limits, error handling, and more.
Under the hood, the HTTP Request destination leverages Hightouch's powerful sync engine, so you continue to benefit from the security and observability features available in all our native destinations.
This article provides a technical overview of how the HTTP Request destination works. Skip to the guided tutorial showing how to create a custom integration with Algolia's REST API if you want to follow along with an example.
It's strictly prohibited to use the HTTP request destination to replicate
functionality already supported by a native destination.
Hightouch queries your data source and monitors the query results for added rows, changed rows, and/or removed rows. When configuring the HTTP Request destination, you specify which of these events should trigger an HTTP request. For each trigger, you then specify an endpoint, payload, and other parameters.
To help Hightouch play nicely with other services, you can also specify constraints such as rate limits, concurrency limits, and error handling logic. You can configure all this in the Hightouch app—no code required.
Let's walk through the process to see how it works.
You can optionally provide headers to include in each outbound HTTP request. These headers often include access tokens used for authentication.
To enter a header, enter its key name in the input. If you want a header property to be obscured in the UI and encrypted in storage, select the Secret checkbox. Click Add key, then enter the key's value in the input that appears to the right. Repeat this for as many headers as you want to add.
If you need to fetch a new token during each sync, consider using
OAuth for authentication. You can also consider using one of our
serverless function destinations, for example, AWS
Lambda.
You can optionally add a custom CA, SSL, or TLS certificate.
Enable the relevant toggle and upload your certificate files.
Certificates can have a maximum size of 10 MB.
We also support OAuth authentication. OAuth implementations can vary between providers, so we recommend using a tool like Postman to test the token refresh process end-to-end. Once you've confirmed that you're able to fetch access tokens and authorize requests using Postman, you can copy your Postman configuration into Hightouch verbatim.
During the OAuth flow, multiple requests are made to retrieve or refresh access tokens. Configure the following
options to include additional headers, URL parameters, or values in the request body.
Authorization request URL parameters: URL parameters sent in the Auth URL for the authorization code grant type.
Token request: Additional parameters and values sent with the request to the Access token URL.
Refresh request: Additional parameters and values sent with the refresh token request to the Refresh token URL or
Access token URL if no refresh URL is provided.
Once you've created your HTTP Request destination and have a model to pull data from, you can set up your sync configuration to begin syncing data. Go to the Syncs overview page and click the Add sync button to begin. Then, select the relevant model and the HTTP Request destination you previously set up.
In this first step, you tell Hightouch when to trigger HTTP requests. You then configure each trigger separately. This lets your integration have different behaviors depending on whether a row is added, changed, and/or removed. Most APIs use different endpoints and HTTP methods for creating, updating, and deleting resources.
For example, suppose you're syncing customer data to an email marketing tool. You might want to create a contact when a row is added, update an existing contact when a row changes, and delete an existing contact when a row is removed. Having separate configuration for each trigger offers the flexibility required to support this use case.
A valid sync configuration requires at least one trigger.
The number of operations displayed on the run details
page only counts the operations for the
request triggers you configured. For example, if you only configured the
Rows added trigger, the run details page doesn't display information
related to changed or removed rows.
You can send a single row or batch of multiple rows in each HTTP request.
To enable batching, select A batch of multiple rows in your sync configuration. The default batch size is 100, but you can enter a different value if needed.
In this step, you tell Hightouch how to send HTTP requests to your internal system, third-party API, or other web service.
Hightouch supports GET, POST, PUT, PATCH, and DELETE requests to any endpoint.
When specifying the endpoint URL, you may either provide a static URL or a dynamic, templated URL.
Recall that you provided the base URL in the initial destination setup, so you only need to provide the specific endpoint path.
If you provide a static URL, Hightouch uses that URL for all HTTP requests triggered by the sync. For example, Stripe's REST API uses a single, static URL for creating new customer objects: POST https://api.stripe.com/v1/customers.
If you provide a dynamic URL, Hightouch generates a different URL for each HTTP request by incorporating data from your model. For example, Stripe's REST API requires you to reference an ID when deleting specific objects. You could incorporate the objectID into the URL template like this: DELETE https://api.stripe.com/v1/customers/{{customer_id}}.
To insert a value from your model, use {{row.column_name}}, where column_name references any column in your model. For example, {{row.first_name}} or {{row.email_address}}.
To change the output of a Liquid object, you can append a filter with a single pipe (|):
{{ row.column_name | filter }}.
For example, let's say the value of the first_name column in a row is "Alice". The Liquid template {{ row.first_name | upcase }} would generate {{ "Alice" | upcase }} → ALICE.
You can also chain multiple filters together like this: {{ row.product_id | remove: "0" | prepend: "SKU_"}} → {{ "000123" | remove: "0" | prepend: "SKU_"}} → SKU_123.
In this step, you tell Hightouch how to build the request payload using data from your model. Hightouch supports JSON and XML payloads, as well as URL encoded forms. You can also send an empty body, if needed.
Most modern web APIs use JSON payloads. This destination offers three methods of composing a JSON request body:
With the JSON editor, you can compose any JSON object using the Liquid template language. This is particularly useful for complex payloads containing nested objects and arrays, which can sometimes be difficult to model entirely in SQL.
Suppose your data model looks like this:
full_name
age
email_address
phone_number
John Doe
30
john@example.com
+14158675309
And you want your HTTP request to have a payload like this:
When batching is enabled, the context is changed from a single record (row) to multiple records (rows).
With multiple rows, you likely want to use loops to build your payload.
Suppose your data model looks like this:
full_name
age
email_address
phone_number
John Doe
30
john@example.com
+14158675309
Jane Doe
29
jane@example.com
+14158674319
And you want your HTTP request to have a payload like this:
Don't worry about unnecessary trailing commas when evaluating loops in your
payload template. Hightouch automatically strips trailing commas to conform
with the JSON specification.
If you're already storing JSON data in your source, or if you have the ability to construct a JSON object using SQL, you can select one column in your model that already contains the full request payload.
This setting is commonly used when syncing web events that have already been collected and stored as JSON objects in your database.
When batching is enabled, the payload is an array where each element is a row value from the selected column. For example, if you select a column that includes a JSON object as each row value, the batch payload would be an array of those JSON objects.
You can use the field mapper to rename fields. For
example, you can map first_name to customer_first_name. The field mapper
offers additional capabilities, such as inline
mapping to create objects and arrays.
In this step, you configure rate limits and concurrency. In other words, you tell Hightouch how to avoid overwhelming your HTTP endpoint with requests.
Most modern web APIs enforce rate limits, which set a maximum allowed number of requests per second, minute, or hour. Occasionally, APIs may also have concurrency limits, which set a maximum allowed number of requests that can be processed simultaneously.
Hightouch defaults to 1000 requests per second or 100 concurrent requests, whichever limit is reached first. You can override these defaults to meet the requirements of your web service. Keep in mind that rate limits and concurrency limits both affect sync speed.
To ensure that Hightouch never exceeds your chosen limits, we apply a small offset to the values provided in the sync configuration form.
Hightouch waits out the entire rate limit when a sync finishes, to make sure the next run has the full number of requests and doesn't violate the rate limit configuration.
For example, if you set the rate limit window to Requests per hour, a sync run waits one full hour before triggering a new run.
If you want to avoid this behavior, you can set the window to Requests per second or Requests per minute.
Concurrency and rate limit controls aren't shared between syncs and don't
persist across sync runs.
In this step, you tell Hightouch how to handle errors, such as HTTP request timeouts and error responses.
These errors will be retried indefinitely until they succeed. You have a choice between retrying immediately or waiting until the next sync run. If you elect to retry immediately, you can specify how many retries should be attempted during the sync run. If all of these retries fail, the request will be retried during subsequent sync runs until it succeeds. Setting up alerts for row and sync errors is always recommended.
Any HTTP response with a 400- or 500-level status code is considered an error.
In this step, you tell Hightouch how to handle rows present in your model results during the first sync run.
Certain workflows, such as hydrating a CRM for the first time, may require performing a backfill of all rows during the initial sync. For other use cases, such as sending product notifications, you may want to skip existing rows and only make HTTP requests for future data changes.
This guided tutorial uses the HTTP Request destination to integrate Hightouch with Algolia's REST API. Specifically, it gives instructions for how to sync synonyms in a search index for each row added to a data model.
Hightouch provides an Algolia integration that allows you to index records and events, but not synonyms, rules, or other Algolia objects.
First, we go to the Destinations overview page and click the Add destination button. Then, we select HTTP Request and click Continue. Next, the UI prompts us to enter a base URL, HTTP headers, and certificate.
Algolia's Search API has the same host URL for searching, adding objects, and adding settings like synonyms.
We can enter this URL as the base URL in Hightouch.
According to Algolia's docs, you should authenticate API requests with these headers:
X-Algolia-Application-Id: the ID of your Algolia application
In Hightouch, we would provide these as the HTTP headers.
Be sure to replace the example headers with your actual Algolia application ID and an API Key with the write permission.
Since we don't need to use any security certificates, we can leave those options turned off and click Continue.
The last step of the initial configuration is giving our destination a descriptive name.
In this case, "HTTP Request - Algolia REST API" lets us know what we can use this destination for.
Here's how we can configure the rows added trigger in Hightouch to match this:
Select to add a batch of multiple rows in each request, we can leave the default batch size of 100
Select POST as the HTTP method
Enter /1/indexes/{indexName}/synonyms/batch as the URL
In the preceding screenshot, the application ID and index name we want to add synonyms to have been filled out; you need to include your own Algolia application ID and index name.
Algolia's docs say that the entire API uses JSON, so we select JSON as the payload type.
To construct the request payload, we need to reference our Hightouch model, which might look like this:
We can construct this expected JSON payload by using the JSON editor and inputing this as the JSON payload definition:
[{% for row in rows %}{"objectID":"{{ row.id }}","type":"synonym","synonyms":["{{ row.synonyms | replace: ', ', '","' }}"],},{% endfor %}]
This example uses the Liquid replace
function to create a
properly formatted array from a comma-separated string. If the data type in
the model results is already an array, this transformation isn't necessary.
Algolia starts throttling indexing operations when you have over 100 pending requests, so we set a rate limit of 100 requests per second and a concurrency limit of 100 requests at a time.
In the unlikely scenario that Algolia's API is experiencing downtime, we would want to wait a while before retrying failed rows. Therefore, we configure our sync to retry errors during the next sync run and give Algolia 30 seconds to respond to each of our HTTP requests.
Lastly, we want to make sure that our first sync run backfills our Algolia index with synonyms for all rows present in the initial model results.
That's it. Using Hightouch's UI, we've built a custom integration with Algolia to create new synonyms whenever new rows appear in our Hightouch model.
Once we've run the sync, we should check our Algolia dashboard to confirm the synonyms appear as we expect.
We can also use the Hightouch debugger to inspect sync runs.
In the future, we might want to update and delete existing synonyms so that our Algolia index always stays in sync with the latest data in our source.
To do this, we would enable the rows changed and rows removed triggers for the HTTP Request destination. The configuration for each of these triggers would be slightly different from the tutorial above, but the same concepts apply. Reference the Algolia API docs to determine the appropriate HTTP method, endpoint, and payload for each operation.
Ready to get started?
Jump right in or a book a demo. Your first destination is always free.