For the complete documentation index, see llms.txt. This page is also available as Markdown.

Receiving Events

A few things to consider when receiving events forwarded by a Data Connector.

Request Contents

Events are delivered to the receiving endpoint as HTTPS POST requests.

Both the headers and body of the incoming request contain information of interest that can be extracted and, depending on the configuration, used to verify the content and origin of the request.

Exploring the request contents

You can use webhook.site to explore the contents of the POST requests. This site will generate a URL that you can point a Data Connector to, and it will display all the relevant details about each request.

Note that using this service will make the events publicly available through the URL generated by that site. We recommend exploring this with emulators in a separate project in DT Studio.

Every forwarded event includes the following header.

  • DT-Asymmetric-Signature Includes a JSON Web Token (JWT) signed by DT with an ECDSA private key (ES256). It is verified using DT's public key, published at a JWKS endpoint. No shared secret is involved. This is the recommended way to verify events.

If a signature secret is set in the Data Connector configuration, the following header will be included.

  • X-Dt-Signature Includes a JWT signed with your Signature Secret using HS256. Supported for existing integrations.

Both tokens carry a checksum of the same request body. Verifying either one is sufficient; if both are present, prefer DT-Asymmetric-Signature.

Note that depending on the framework used to receive the Data Connector events, the header name casing may differ. Some services will force header names to be lower-cased, like x-dt-signature.

Read more about validating the request signature in the Verifying Signed Events section below.

Body

The request body contains three fields, event, labels, and metadata. The following snippet shows an example request body of a touch event for a humidity sensor forwarded by a Data Connector.

Field

Type

Description

event

struct

Contains event data. See the Event documentation where the structure is explained in detail for each event type.

labels

struct

Device label key- and value pairs included by the Data Connector. See the Advanced Configuration page for details about including labels.

metadata

struct

Contains metadata about the device that is the source of the event. See the section below for more details.

Event Metadata

Each event has a metadata field that includes details about the device that is the source of the event. The event metadata has the following structure.

Field
Type
Description

deviceId

string

The identifier of the device that published the event.

projectId

string

The identifier of the project the device is in.

deviceType

string

The device type that published the event.

productNumber

string

The product number of the device that published the event.

Note

The structure of the metadata field might change in the future if event types are added that are not published by devices. Make sure to first check event.eventType to make sure it is a known device event before processing the metadata field.

See the code sample below for an example of how to do this.

The event metadata makes it possible to check which device type has published the event, even for event types like touch or networkStatus which are published by many types of devices. This makes it possible to add new devices to a database without having to first look up the device using the REST API.

The metadata also provides a more convenient way to get the deviceId and projectId of the device that published the event, without having to parse the event.targetName field.

Implementing Your Endpoint

In order to receive events from a Data Connector, your server needs to be set up to do the following:

  • Start listening for incoming requests on the URL specified in the Data Connector's Endpoint URL.

  • Read the HTTP headers and body, and process the event.

  • Reply with a 2XX status code in a timely manner. Any response code outside the 2xx range will be considered a failed delivery and will be retried.

If your endpoint fails to reply with status codes in the 2xx range consistently for an extended time period, the Data Connector will eventually be automatically disabled.

This server can be written in any programming language. Example implementations in a selection of languages can be found on the Example Integrations page, as well as in the Verifying Signed Events section on this page.

Regardless of which language is used to implement the server, there are a few things to keep in mind:

Server Configuration

The server needs to be set up to listen for HTTPS POST requests on the URL specified in the Data Connector's Endpoint URL. This endpoint needs to be publicly available and not require any authorization. You can verify that the event originates from DT by following the steps in the Verifying Signed Events section.

The server also needs to be configured with a valid SSL certificate that is issued from one of the root certificates from Mozilla's CA Certificate Program. This will be the case for most certificates (e.g. certificates issued by Let's Encrypt, DigiCert, GlobalSign, etc). Self-signed certificates are not supported.

SSL Certificate Verification

You can verify that you have a valid SSL cert by running curl -v {YOUR_ENDPOINT} in your terminal and look for the string "SSL certificate verify ok".

To verify that your SSL cert is issued by one of the root certs in Mozilla's CA Certificate Program, you can spin up an Alpine docker container locally, and check your SSL cert from that container. Alpine uses the root certs in Mozilla's CA Certificate Program to validate SSL certificates. Run the following commands in your terminal (assuming Docker is installed):

  1. docker run --rm -ti alpine:latest sh

  2. apk --update add ca-certificates curl

  3. curl -v {YOUR_ENDPOINT}

  4. Make sure the "SSL certificate verify ok" string is present in the output

  5. Use ctrl+d to quit the container

To help diagnose any SSL issues, you can use the SSL Server Test from SSL Labs.

Handle Incoming Event

Each event will be delivered to your endpoint separately as individual requests. When processing a request, you should do the following steps:

  1. (Recommended) Verify the integrity of the event, and that it originates from DT (see Verifying Signed Events below).

  2. Do something with the event. You might for example put in on a separate queue for later processing (Google Pubsub, Amazon SQS, Azure ServiceBus, etc), write it to a database, or do some other processing.

  3. Respond with a status code in the 2xx range once you've accepted/processed the event.

These steps should all be done in a timely manner, and respond within 10 seconds. If your endpoint takes more than 10 seconds to respond, you run the risk that it will time out and be retried according to the Retry Policy.

If you think there's a possibility that your processing might take more than 10 seconds, prefer to write the event to a separate queue or a database and do the processing at a later point in time.

Handling Duplicates

Every event received by DT Cloud is put in a dedicated, per-Data Connector queue. Messages are removed from this queue once acknowledged, or if the message is older than 12 hours.

A side effect of this delivery guarantee is that, under certain conditions, you may receive duplicates of the same event. While rare, deduplication should be performed on the receiving end by checking event IDs.

Retry policy

Any time a Data Connector does not receive a successful response (HTTP status code 2xx), the event will be retried. If an event has not been successfully acknowledged after 12 hours, it will be discarded.

The retry interval is calculated as an exponential backoff policy, given by

t0⋅2n−1,t_0\cdot2^{n-1},

where t0t0 is the initial interval of 8 seconds and nn the attempt number. The interval will not exceed 1 hour. For very slow endpoints, the minimum retry interval will be 4x4x the response time.

The following table shows the retry interval after a given number of delivery attempts:

Attempt

Retry Interval [s]

1

8

2

16

3

32

...

...

9

2048

10

3600

11

3600

Verifying Signed Events

A signed Data Connector event lets you verify two things before you process it: that the request actually originated from DT Cloud, and that the body has not been modified in transit. In outline, both signing methods work the same way. A JWT in an HTTP header contains a checksum of the request body. What differs is how the token is signed and verified.

Asymmetric (recommended)
Symmetric

Availability

Included on every event, no configuration needed

Only when a Signature Secret is configured

Header

DT-Asymmetric-Signature

X-Dt-Signature

Algorithm

ES256 (ECDSA)

HS256 (HMAC)

Verified with

DT's public key, fetched from a JWKS endpoint

Your Signature Secret

Shared secret to store and rotate

No

Yes

Identifies the sending data-connector and organization

Yes, via the sub and organization_id claims

No

DT signs each event with a private key that never leaves DT Cloud, and you verify it with the corresponding public key. Because the verifying key is public, there is no secret for your endpoint to store, no secret to leak, and no secret to rotate. We recommend asymmetric verification for all integrations.

Note

A Signature Secret remains available as an opt-in configuration, and existing integrations using one continue to work unchanged. Configuring a secret does not disable asymmetric signing. Events will carry both headers, and verifying either one is sufficient. See Symmetric verification with a Signature Secret below

Asymmetric Verification

DT publishes its signing keys through an OpenID Connect discovery document:

The jwks_uri field in that document points to the JSON Web Key Set containing the public keys. Most languages have an OIDC or JWKS library that handles discovery, key lookup, and caching for you. We recommend using a library rather than implementing the verification by hand.

Cache the key set

Fetching the discovery document and JWKS on every request adds latency to each event and may get you rate limited. Use a library that caches the key set, or cache it yourself. Do not hardcode a public key. Signing keys are rotated, and your verification will break when they are. Always resolve the key by the kid in the token header.

The token contains the following claims.

Claim
Type
Description

iss

string

The issuer, always https://identity.disruptive-technologies.com/data-connector.

sub

string

The resource name of the Data Connector that sent the event, e.g. projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>.

iat

number

Unix timestamp for when the token was issued.

exp

number

Unix timestamp for when the token expires.

organization_id

string

The identifier of the DT organization the sending Data Connector belongs to.

checksum_sha256

string

Hex-encoded SHA256 checksum of the request body.

The steps to verify a received request are as follows.

  1. Extract the signed JWT from the DT-Asymmetric-Signature header of the received request.

  2. Look up the signing key for the JWT token's kid header field in DT's JWKS, and verify the token's signature. Also verify the iss, exp, and iat claims. Data Connector events are always signed with ES256. Configure your verifier to accept that algorithm and no other.

  3. Verify that the event came from a sender you expect, using either the sub or the organization_id claim. See the note below on choosing between them.

  4. Calculate a SHA256 checksum over the raw request body.

  5. Compare that checksum with the checksum_sha256 claim in the JWT.

If the signature is valid and the checksums are identical, the event originated from a Data Connector in your organization and has not been tampered with.

Choosing between sub and organization_id

sub tells you which Data Connector sent the event. organization_id tells you which organization it belongs to. Check whichever fits your setup:

  • One endpoint, one Data Connector. Compare sub to that connector's resource name. This is the strictest check.

  • Your own code creates the Data Connectors. If you create them through the REST API, save each resource name and check sub against that list. Your code creates them, so the list stays up to date. Events from any other connector are rejected, including ones added by hand in DT Studio.

  • You don't control which connectors send events. Compare organization_id to your organization ID. Checking sub is stricter, but a list of connector names breaks when someone adds a connector you don't know about.

Check at least one. A valid signature and checksum only prove the event came from DT, not that it came from a connector you set up.

Symmetric Verification with Signature Secret

Note

New integrations should prefer asymmetric verification instead. This method is documented for existing integrations.

The following steps sum up the process of verifying the received request at the receiving endpoint.

  1. Extract the signed JWT from the HTTP header X-Dt-Signature of the received request.

  2. Verify the JWT's HMAC signature with the signature secret.

  3. Calculate a SHA256 checksum over the entire request body.

  4. Compare the body checksum with the checksum_sha256 field contained in the JWT (there's a checksum field as well that uses SHA1 which is less secure than SHA256, and is kept only for backward compatibility).

  5. If these checksums are identical, you can be certain that the event has not been tampered with and originated from your Data Connector.

The following snippet from our Google Cloud Function example integration implements this verification process.

Last updated

Was this helpful?