# Getting Started

A quick overview of the tools and services available for developers on our platform. To get you started quickly, a short summary of core concepts is provided to point you in the right direction.

## Devices

Disruptive Technologies (DT) aims to make the process of gathering large quantities of data as simple as possible. Our wireless sensors send new events to a Cloud Connector that propagates them to our cloud, where you can access the data using either DT Studio or our various APIs. As the sensors securely communicate with all available Cloud Connectors within range, no setup or pairing is required.

![](https://storage.googleapis.com/gitbook-assets/DTSensorFlow.gif)

Head over to the [Devices](/concepts/devices) page to get an overview of the wide selection of devices we offer. 

### Sensor Emulator

If you do not have access to physical devices yet, our Sensor Emulator API lets you create and interact with emulated devices that behave just like real ones. This can be a valuable tool for prototyping, debugging, or simply for exploring our various services.

Head over to the [Sensor Emulator](/sensor-emulator) page to get started.

## Two Integration Paths

There are two paths to integrate with our platform. Data Connectors pushes events directly to your Cloud Service, and the REST API lets you control everything programmatically.

![](/files/-MUAgsDlcOjW62UTBAS_)

### Data Connectors

A simple yet reliable method of forwarding your data into an external service. They work like webhooks by sending device events as HTTPS POST requests, with additional features for security and reliability that make them perfect for the following integrations.

* Long-term storage in your own database.
* Data science applications that require a constant stream of data.
* Triggering alarms from data events.

Read the [Introduction to Data Connectors](/data-connectors/introduction-to-data-connector) to get started.

### REST API

Our REST API can be used to interact with our service programmatically. For instance, DT Studio, our easy-to-use device management web app, is itself entirely built upon the REST API. Therefore, everything you can do in DT Studio, from moving devices between projects and managing Service Accounts, can also be done via our REST API.

Read the [Introduction to REST API](/introduction-to-rest-api) to get started.

## Service Accounts

All programmatic interaction with our APIs is done via a logged-in Service Account. Once a client is authenticated, the program interacting with the API can perform operations within the role access level granted to the Service Account.

Read the [Introduction to Service Accounts](/service-accounts/introduction-to-service-accounts) to get started.

## Application Notes

Our sensors are designed to let you gather data from almost any environment, which opens the door for many great use-cases. While DT does not currently deliver end-user solutions for data science applications, our internal developers have written a few application notes where some are explored, implemented, and discussed.

Head over to the [Application Notes](/other/application-notes) page to read about some of the ideas we have for using our sensors.


# Overview

An overview of important concepts and hierarchies you will encounter when using our services.

We have created a flexible yet simple hierarchy for organizing devices into projects. The figure below shows how the various services are interconnected, each of which is explained in the following sections.

<div align="center"><img src="/files/jrcRNRwtWL97YRwFqGoZ" alt=""></div>

## Organizations

The organization is the top level in the hierarchy and represents the legal entity that owns the sensors and is invoiced for device subscriptions. This could be an organization, a department within an organization, or an individual.

### Members

Organizations and projects can have members with different [roles](/service-accounts/managing-access-rights#roles). A member can be either a User or a [Service Account](/service-accounts/introduction-to-service-accounts). By default, the person who creates an organization is made an Organization Administrator who can create new projects and invite new members.

## Projects

Projects are used within an organization to create a collection of devices. The intention is to make it easier for users to get an overview and manage access control of select devices. An Organization Admin can create as many new projects as they see fit, and they can add and grant new members access to specific projects.

### Inventory Project

Although users can create as many projects as they want, every organization starts with a default project called Inventory.

The Inventory Project has a unique badge in [DT Studio](https://studio.disruptive-technologies.com/) and has the boolean `inventory` flag set when listed using our [REST API](https://developer.disruptive-technologies.com/api#/Organizations%20%26%20Projects/get_projects).

### Member

A Project Member is a user or a [Service Account](/service-accounts/introduction-to-service-accounts) with access to the devices in a project. A role determines the access level for the user or Service Account and must be assigned to each new Project Member that is created. See [roles](/service-accounts/managing-access-rights#roles) for more information about the types of roles available.

### Service Accounts

Service Accounts are used to give other services access to projects through the use of REST APIs. While a user represents a person and is used for logging in to DT Studio, Service Accounts represent services like programs running on another server. See our guide on [Creating a Service Account](/service-accounts/creating-a-service-account).

![](/files/-MTehPLyB4lfniqHgYfd)

Once a Service Account is created, it has to be added as a member to one or more Projects or Organizations and assigned a [Role](/service-accounts/managing-access-rights#roles) that determines what it is allowed to view or edit.&#x20;

#### Keys

Keys are credentials used by an application to authenticate the Service Account against our cloud service. A Service Account can be issued multiple keys that can be used by one or several services.&#x20;

### Data Connectors

Data Connectors are used to reliably send data from sensors in a project to a 3rd party service. A project can have multiple Data Connectors, and the data sent through each one can be configured using [DT Studio or our REST API](/data-connectors/advanced-configurations).

![](/files/-MTaZT9NJ-VF990L7BFo)

## Devices

[Device](/concepts/devices) is a term used to describe Wireless Sensors, Cloud Connectors, and emulated devices.

### Events

[Event](/concepts/events) is a term used to describe a collection of data sent between services. This is primarily sensor measurements such as temperature and other things such as when the labels for a device change. All event types will have the same [base JSON structure](/concepts/events#event-structure), and the data related to the specific event is represented in the `data` field found within.

### Device Labels

Labels are key-value pairs that can be assigned as metadata to a device through DT Studio or using the REST API. Each device can have multiple labels, and they can be used as a filter when [listing devices](https://developer.disruptive-technologies.com/api#/Devices%20%26%20Labels/get_projects__project__devices) or setting up [event streams](/examples/streaming-events). They can also be included in the JSON structure for the [events that are pushed](/data-connectors/advanced-configurations#including-labels) to Data Connectors.

Labels are a great way to group and organize devices within a project. Typical use-cases for labels include specifying which floor or building the devices are installed in, the current status of a device, such as if it is currently being installed or not, or a tag to make it easier to search for it in 3rd party services.

![](/files/-MVuSn6VBDhVLcZ0dFU9)

#### DT Studio Specific Labels

DT Studio uses labels with the keys `name` and `description` to show as the display name and the description of a device. It is recommended that an integration uses these labels in the same way when displaying or updating a device.


# Devices

An overview of devices, where they can be found, and the various types.

## Device Types

The following table provides an overview of all the device types that can be found throughout our API.

<table data-header-hidden><thead><tr><th>Device Name</th><th width="194.3522293381693">Device Type</th><th>Description</th></tr></thead><tbody><tr><td><strong>Device Name</strong></td><td><strong>Device Type</strong></td><td><strong>Description</strong></td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#temperature-sensor">Temperature</a></td><td><code>temperature</code></td><td>Sensor that reports temperature in Celsius.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#proximity-sensor">Proximity</a></td><td><code>proximity</code></td><td>Sensor that reports objects in close proximity.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#touch-sensor">Touch</a></td><td><code>touch</code></td><td>Sensor that reports when the sensor is touched.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#humidity-sensor">Humidity</a></td><td><code>humidity</code></td><td>Sensor that reports temperature in Celsius and relative humidity as a percentage.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#counting-proximity-sensor">Counting Proximity</a></td><td><code>proximityCounter</code></td><td>Sensor that reports total count of proximity events.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#counting-touch-sensor">Counting Touch</a></td><td><code>touchCounter</code></td><td>Sensor that reports total count of touch events.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#water-detector-sensor">Water Detector</a></td><td><code>waterDetector</code></td><td>Sensor that reports whether or not water is present.</td></tr><tr><td><a href="#co2-sensor">CO2</a></td><td><code>co2</code></td><td>Sensor that reports CO2, Barometric Pressure, Temperature in Celsius, and Relative Humidity.</td></tr><tr><td><a href="/pages/-MToY47tBuqyLn0Cn6cs#motion-sensor">Motion</a></td><td><code>motion</code></td><td>Sensor that reports when motion is detected and when it's no longer detected.</td></tr><tr><td><a href="#desk-occupancy-sensor">Desk Occupancy</a></td><td><code>deskOccupancy</code></td><td>Sensor that reports when a desk is occupied and when it's no longer occupied.</td></tr><tr><td><a href="#contact-sensor">Contact</a></td><td><code>contact</code></td><td>Sensor that reports whether a door/window is open or closed.</td></tr><tr><td><a href="#cloud-connector-details">Cloud Connector</a></td><td><code>ccon</code></td><td>Connects sensors securely to the cloud.</td></tr></tbody></table>

![](/files/-MTooyTDz8WGDYF2pgFI)

## Structure

A device is either a sensor or a Cloud Connector and has a defined structure in our APIs. They can be fetched either by [listing the devices in a project](https://developer.disruptive-technologies.com/api/#/Devices%20%26%20Labels/get_projects__project__devices) or [looking up a single device](https://developer.disruptive-technologies.com/api/#/Devices%20%26%20Labels/get_projects__project__devices__device_), both of which will return device(s) with the structure as shown in the snippet below.

```javascript
{
    "name": "projects/<PROJECT_ID>/device/<DEVICE_ID>",
    "type": "<DEVICE_TYPE>",
    "productNumber": "<PRODUCT_NUMBER>",
    "labels": {
        "<KEY>": "<VALUE>",
        ...
    },
    "reported": {
        ...
    }
}
```

| Field           | Type     | Description                                                                                                                                                                                                                                                                                                             |
| --------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`          | `string` | The device resource name. Includes both the identifier of the project the device is currently in and the identifier of the device itself.                                                                                                                                                                               |
| `type`          | `string` | The device type. See the [overview](/concepts/devices#wireless-sensors) below for a full list.                                                                                                                                                                                                                          |
| `productNumber` | `string` | The device product number. This is the same product number that can be found on the [Sensors](https://support.disruptive-technologies.com/hc/en-us/categories/4404697444882-Sensors) and [Cloud Connectors](https://support.disruptive-technologies.com/hc/en-us/sections/360003168340-Cloud-Connectors) support pages. |
| `labels`        | `struct` | Device labels are user-configurable metadata key-value pairs. DT Studio uses labels with the keys `name` and `description` for the display name and description of the device, respectively.                                                                                                                            |
| `reported`      | `struct` | Contains the last [Event](/concepts/events) of each event type received for this device. Which events are included here depends on the device type. See the [overview](/concepts/devices#wireless-sensors) below to learn which event type is included with which device type.                                          |

## Wireless Sensors Details

The following provides details of all the types of wireless sensors found throughout our APIs. Each type may contain one- or several different product numbers.&#x20;

### Temperature Sensor

Reports temperature in Celsius.

* **Device Type:** `temperature`
* **Variations:**
  * [Temperature Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360010342900-Temperature-Sensor)
  * [Temperature Sensor (2nd Gen)](https://support.disruptive-technologies.com/hc/en-us/articles/4405644706194-Temperature-Sensor-2nd-Gen-)
  * [Temperature Sensor (3rd Gen)](https://support.disruptive-technologies.com/hc/en-us/articles/9389983018268-Temperature-Sensor-3rd-Gen-)
  * [Industrial Temperature Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360019263639-Industrial-Temperature-Sensor-)
  * [Industrial Temperature Sensor (2nd Gen)](https://support.disruptive-technologies.com/hc/en-us/articles/4413995314194-Industrial-Temperature-Sensor-2nd-Gen-)
  * [Temperature Sensor EN12830/330s](https://support.disruptive-technologies.com/hc/en-us/articles/360010452139-Temperature-Sensor-EN12830-330s)
  * [Temperature Sensor (3rd Gen)](https://support.disruptive-technologies.com/hc/en-us/articles/9389983018268-Temperature-Sensor-3rd-Gen-)
  * [Temperature Sensor with Data Backfill](https://support.disruptive-technologies.com/hc/en-us/articles/9389983018268-Temperature-Sensor)
  * [Temperature Probe Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/11432439857948-Wireless-Temperature-Probe-Sensor)

The following event types are available for this device type.

| Event Name                                                    | Event Type        | Trigger                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Temperature](/concepts/events#temperature-event)             | `temperature`     | <p>Every periodic heartbeat or when touched.<br><br><strong>Note:</strong> <a href="https://support.disruptive-technologies.com/hc/en-us/articles/4405644706194-Temperature-Sensor-2nd-Gen-">2nd Gen Temperature Sensors</a> and <a href="https://support.disruptive-technologies.com/hc/en-us/articles/9389983018268-Temperature-Sensor-3rd-Gen-">3rd Gen Temperature Sensor</a> will only send temperature on heartbeats, and not when touched.</p> |
| [Touch](/concepts/events#touch-event)                         | `touch`           | <p>When touched. </p><p></p><p>This event is not supported by the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/11432439857948-Wireless-Temperature-Probe-Sensor">Temperature Probe Sensor</a>.</p>                                                                                                                                                                                                                          |
| [Network Status](/concepts/events#network-status-event)       | `networkStatus`   | Every [periodic heartbeat](/concepts/events#periodic-heartbeat).                                                                                                                                                                                                                                                                                                                                                                                      |
| [Battery Status](/concepts/events#battery-status-event)       | `batteryStatus`   | Approximately once per day.                                                                                                                                                                                                                                                                                                                                                                                                                           |
| [Probe Wire Status](/concepts/events#probe-wire-status-event) | `probeWireStatus` | <p>When the wires are plugged in or unplugged from a <a href="https://support.disruptive-technologies.com/hc/en-us/articles/11432439857948-Wireless-Temperature-Probe-Sensor">Temperature Probe Sensor</a>.<br><br>This event is only supported by the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/11432439857948-Wireless-Temperature-Probe-Sensor">Temperature Probe Sensor</a>.</p>                                     |

### Proximity Sensor

Reports objects in close proximity.

* **Device Type:** `proximity`
* **Variations:**
  * [Proximity Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360010452099-Proximity-Sensor)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [Object Present](/concepts/events#object-present-event) | `objectPresent` | When an object appears or disappears in close proximity.         |
| [Touch](/concepts/events#touch-event)                   | `touch`         | When touched.                                                    |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

### Touch Sensor

Reports when the sensor is touched.

* **Device Type:** `touch`
* **Variations:**
  * [Touch Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360010344980-Touch-Sensor)
  * [Tactile Touch Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360010452079-Wireless-Tactile-Touch-Sensor)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [Touch](/concepts/events#touch-event)                   | `touch`         | When touched.                                                    |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

### Humidity Sensor

Reports temperature in Celsius and percentage humidity.

* **Device Type:** `humidity`
* **Variations:**
  * [Humidity Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360013965460-Wireless-Humidity-Sensor)

The following event types are available for this device type. Note that temperature measurements are found within the [Humidity](/concepts/events#humidity-event) event.

| Event Name                                              | Event Type      | Trigger                                                                          |
| ------------------------------------------------------- | --------------- | -------------------------------------------------------------------------------- |
| [Humidity](/concepts/events#humidity-event)             | `humidity`      | Every [periodic heartbeat](/concepts/events#periodic-heartbeat) or when touched. |
| [Touch](/concepts/events#touch-event)                   | `touch`         | When touched.                                                                    |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat).                 |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                                      |

### Counting Proximity Sensor

Reports the total number of proximity state changes on each heartbeat. The counter does not reset on heartbeats and will keep getting incremented. The presence of an object and the absence of an object are counted individually, and will both increment the counter with 1. A counting proximity sensor also reports normal touch events if touched and held for a few seconds. This allows the sensor to be identified through [Touch to Identify](/examples/touch-to-identify).

* **Device Type:** `proximityCounter`
* **Variations:**
  * [Counting Proximity Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360011443399-Counting-Proximity-Sensor)

The following event types are available for this device type.

| Event Name                                                          | Event Type           | Trigger                                                          |
| ------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------- |
| [Object Present Count](/concepts/events#object-present-count-event) | `objectPresentCount` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Touch](/concepts/events#touch-event)                               | `touch`              | When touched and held for a few seconds.                         |
| [Network Status](/concepts/events#network-status-event)             | `networkStatus`      | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event)             | `batteryStatus`      | Approximately once per day.                                      |

### Counting Touch Sensor

Reports the total number of touches on each heartbeat. The counter does not reset on heartbeats and will keep getting incremented. A counting touch sensor also reports normal touch events if touched and held for a few seconds. This allows the sensor to be identified through [Touch to Identify](/examples/touch-to-identify).

* **Device Type:** `touchCounter`
* **Variations:**
  * [Counting Touch Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360010783260-Counting-Touch-Sensor)
  * [Counting Tactile Touch Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360011322240-Wireless-Counting-Tactile-Touch-Sensor)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [Touch Count](/concepts/events#touch-count-event)       | `touchCount`    | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Touch](/concepts/events#touch-event)                   | `touch`         | When touched and held for a few seconds.                         |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

### Water Detector Sensor

Reports when water is detected on the surface of the sensor, as well as when it's no longer detected.

* **Device Type:** `waterDetector`
* **Variations:**
  * [Water Detector Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/360012075360-Wireless-Water-Detector)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                                     |
| ------------------------------------------------------- | --------------- | --------------------------------------------------------------------------- |
| [Water Present](/concepts/events#water-present-event)   | `waterPresent`  | When water is detected and no longer detected on the surface of the sensor. |
| [Touch](/concepts/events#touch-event)                   | `touch`         | When touched and held for a few seconds.                                    |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat).            |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                                 |

### CO2 Sensor

Reports CO2, Barometric Pressure, Relative Humidity, and Temperature in Celsius.

* **Device Type:** `co2`
* **Variations:**
  * [CO2 Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/4463537659292-CO2-Sensor)

The following event types are available for this device type. Note that temperature measurements are found within the [Humidity](/concepts/events#humidity-event) event.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [CO2](/concepts/events#co2-event)                       | `co2`           | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Pressure](/concepts/events#pressure-event)             | `pressure`      | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Humidity](/concepts/events#humidity-event)             | `humidity`      | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

### Motion Sensor

Reports motion in the sensor's field of view.

* **Device Type:** `motion`
* **Variations:**
  * [Motion Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/4467893712156-Motion-Sensor)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [Motion](/concepts/events#motion-event)                 | `motion`        | The presence of people in the motion sensor's field of view.     |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

### Desk Occupancy Sensor

Reports when a desk is occupied and when it's no longer occupied.

* **Device Type:** `deskOccupancy`
* **Variations:**
  * [Desk Occupancy Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/5667520055964-Desk-Occupancy-Sensor)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [Desk Occupancy](/concepts/events#desk-occupancy-event) | `deskOccupancy` | When the occupancy status of the desk changes.                   |
| [Touch](/concepts/events#touch-event)                   | `touch`         | When touched.                                                    |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

### Contact Sensor

Reports whether a door/window is open or closed.

* **Device Type:** `contact`
* **Variations:**
  * [Door & Window Sensor](https://support.disruptive-technologies.com/hc/en-us/articles/11433250566940-Wireless-Door-Window-Sensor)

The following event types are available for this device type.

| Event Name                                              | Event Type      | Trigger                                                          |
| ------------------------------------------------------- | --------------- | ---------------------------------------------------------------- |
| [Contact](/concepts/events#contact-event)               | `contact`       | When a door or window is closed or opened.                       |
| [Network Status](/concepts/events#network-status-event) | `networkStatus` | Every [periodic heartbeat](/concepts/events#periodic-heartbeat). |
| [Battery Status](/concepts/events#battery-status-event) | `batteryStatus` | Approximately once per day.                                      |

## Cloud Connector Details

Securely relays device events to our DT Cloud.

* **Device Type:** `ccon`
* **Variations:**
  * [Cloud Connector (2nd Gen)](https://support.disruptive-technologies.com/hc/en-us/articles/6962229244828-Cloud-Connector-2nd-Gen-)
  * [Cloud Connector US 4G](https://support.disruptive-technologies.com/hc/en-us/articles/360014600359-Cloud-Connector-US-4G)
  * [Cloud Connector US Ethernet Only](https://support.disruptive-technologies.com/hc/en-us/articles/360011132779-Cloud-Connector-US-Ethernet-only-)
  * [Cloud Connector EU 4G](https://support.disruptive-technologies.com/hc/en-us/articles/360011013300-Cloud-Connector-EU-4G)
  * [Cloud Connector EU 3G/2G](https://support.disruptive-technologies.com/hc/en-us/articles/360010345040-Cloud-Connector-EU-3G-2G)
  * [Cloud Connector EU Ethernet Only](https://support.disruptive-technologies.com/hc/en-us/articles/360014024259-Cloud-Connector-EU-Ethernet-only-)
  * [Developer Kit](https://support.disruptive-technologies.com/hc/en-us/articles/360013574560-Developer-Kit-Cloud-Connector)

The following event types are available for this device type.

| Event Name                                                    | Event Type         | Trigger                               |
| ------------------------------------------------------------- | ------------------ | ------------------------------------- |
| [Connection Status](/concepts/events#connection-status-event) | `connectionStatus` | When the connection changes.          |
| [Ethernet Status](/concepts/events#ethernet-status-event)     | `ethernetStatus`   | When the ethernet connection changes. |
| [Cellular Status](/concepts/events#cellular-status-event)     | `cellularStatus`   | When the cellular connection changes. |
| [Touch](/concepts/events#touch-event)                         | `touch`            | When touched.                         |


# Events

An overview of the various event types that can be encountered throughout our services.

## Event Types

The following table lists all event types and the triggers that generate them. An overview of event types per device can be found on the [Devices](/concepts/devices) page.

<table data-header-hidden><thead><tr><th>Event Name</th><th width="221">Event Type</th><th>Trigger</th></tr></thead><tbody><tr><td>Event Name</td><td>Event Type</td><td>Trigger</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#touch-event">Touch</a></td><td><code>touch</code></td><td>When touched.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#temperature-event">Temperature</a></td><td><code>temperature</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a> and when touched.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#object-present-event">Object Present</a></td><td><code>objectPresent</code></td><td><code>PRESENT</code>/<code>NOT_PRESENT</code> state change.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#humidity-event">Humidity</a></td><td><code>humidity</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a> and when touched.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#object-present-count-event">Object Present Count</a></td><td><code>objectPresentCount</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a>.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#touch-count-event">Touch Count</a></td><td><code>touchCount</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a>.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#water-present-event">Water Present</a></td><td><code>waterPresent</code></td><td><code>PRESENT</code>/<code>NOT_PRESENT</code> state change.</td></tr><tr><td><a href="#co2-event">CO2</a></td><td><code>co2</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a>.</td></tr><tr><td><a href="#co2-event-1">Pressure</a></td><td><code>pressure</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a>.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#motion-event">Motion</a></td><td><code>motion</code></td><td><code>MOTION_DETECTED</code> / <code>NO_MOTION_DETECTED</code> state change.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#desk-occupancy-event">Desk Occupancy</a></td><td><code>deskOccupancy</code></td><td><code>OCCUPIED</code> / <code>NOT_OCCUPIED</code> state change.</td></tr><tr><td><a href="#contact-event">Contact</a></td><td><code>contact</code></td><td><code>OPEN</code> / <code>CLOSED</code> state change.</td></tr><tr><td><a href="#probe-wire-status-event">Probe Wire Status</a></td><td><code>probeWireStatus</code></td><td>Probe wire state change.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#battery-status-event">Battery Status</a></td><td><code>batteryStatus</code></td><td>See event details.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#network-status-event">Network Status</a></td><td><code>networkStatus</code></td><td>Every <a href="/pages/-MR5VndbmI4ns9Ey_BE2#periodic-heartbeat">periodic heartbeat</a> and together with trigger-based events.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#labels-changed-event">Labels Changed</a></td><td><code>labelsChanged</code></td><td>When a device label is created, modified, or removed.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#connection-status-event">Connection Status</a></td><td><code>connectionStatus</code></td><td>Changes in connection type.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#ethernet-status-event">Ethernet Status</a></td><td><code>ethernetStatus</code></td><td>Changes in the ethernet connection.</td></tr><tr><td><a href="/pages/-MR5VndbmI4ns9Ey_BE2#cellular-status-event">Cellular Status</a></td><td><code>cellularStatus</code></td><td>Changes in the cellular connection.</td></tr></tbody></table>

## Structure

All event types available through our API use the same outer layer structure, shown in the snippet below, where event metadata is found. Information specific to each [event type](/concepts/events#types) is found in the `data` field.

```javascript
{
    "eventId": "<EVENT_ID>",
    "targetName": "projects/<PROJECT_ID>/devices/<DEVICE_ID>",
    "eventType": "<EVENT_TYPE>",
    "data": {
            "See each event type individually below."
        }
    },
    "timestamp": "2024-12-06T13:15:01.576753Z"
}
```

<table data-header-hidden><thead><tr><th>Field</th><th width="194.33333333333331">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>eventId</code></td><td><code>string</code></td><td>The unique identifier of the Event. This can be used to deduplicate events in case the same event is published multiple times.</td></tr><tr><td><code>targetName</code></td><td><code>string</code></td><td>The resource name of the source device. Includes both the identifier of the project the device is currently in, and the identifier of the device itself.</td></tr><tr><td><code>eventType</code></td><td><code>string</code></td><td>The <a href="/pages/-MR5VndbmI4ns9Ey_BE2#event-types">event type</a>.</td></tr><tr><td><code>data</code></td><td><code>struct</code></td><td>Includes the event data and is specific for each <code>eventType</code>. See the <a href="/pages/-MR5VndbmI4ns9Ey_BE2#event-types">event types</a> below for a detailed description of each field that can be found here.</td></tr><tr><td><code>timestamp</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

There are four places where events can be encountered in our services.

1. The `reported` field of a [Device](/concepts/devices#device-structure) when [listing devices through REST API](https://developer.disruptive-technologies.com/api#/Devices%20%26%20Labels/get_projects__project__devices).
2. When pushed to a client using the [event stream REST API endpoint](https://developer.disruptive-technologies.com/api#/Live%20Events%20Streaming%20API/get_projects__project__devices_stream).
3. When fetching historical data through the [event history REST API endpoint](https://developer.disruptive-technologies.com/api#/Event%20History/get_projects__project__devices__device__events).
4. When pushed by a [Data Connector](/data-connectors/introduction-to-data-connector) to another server.

### Periodic Heartbeat

All sensors send a [Network Status](/concepts/events#network-status-event) event at a predefined interval depending on sensor type. Depending on the sensor type, this heartbeat may be accompanied by additional events. For instance, the [Temperature sensor](/concepts/devices#temperature-sensor) will also send a [Temperature event](/concepts/events#temperature-event) every heartbeat.

![](/files/-MTalb-ddOy0UsSIVjyp)

### Triggered Events

A few of our sensors are designed to send events immediately at some trigger. These are sent in addition to, but independent of, the [periodic heartbeat](/concepts/events#periodic-heartbeat).

* [Object Present Event](#object-present-event) for [Proximity Sensors](/concepts/devices#proximity-sensor).
* [Water Present Event](#water-present-event) event for [Water Detector Sensors](/concepts/devices#water-detector-sensor).
* [Touch Event](#touch-event) for [Touch Sensors](/concepts/devices#touch-sensor).
* [Motion Event](#motion-event) for Motion Sensors.
* [Desk Occupancy Event](#desk-occupancy-event) for Desk Occupancy Sensors.

A [Network Status Event](/concepts/events#network-status-event) accompanies each triggered event.

![](/files/-MUmNQsXmWv-fivGJLi8)

### Timestamps

Here are a few notes regarding timestamps and how they are presented in our services.

#### Timezone Offset

All timestamps fetched using our APIs are always given in UTC and must be accounted for. DT Studio will, however, always shows timestamps depending on user locale.

#### Format

The [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) format with [fractional seconds](https://datatracker.ietf.org/doc/html/rfc3339#section-5.3) is used for our timestamps. This is an internet-specific profile of the [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) specification, and an example can be seen below.

* `2024-12-06T08:13:15.361624Z`.

#### Variations

A result of how our events are structured is that two different timestamp fields can be found in each event when fetching them through our API. The `timestamp` and `updateTime` fields are located in the outer layer and `data` field of an event respectively, but will always contain the same value.&#x20;

You may use either `timestamp` or `updateTime` when deciding when an event was received.

A third type, `sampleTime`, can only be found within the [Temperature Event](/concepts/events#temperature-event). These represent the timestamps of inter-heartbeat samples and are estimated by our cloud. [Read more](/concepts/usage-examples/2.gen-temperature-sensor#sample-timestamp-accuracy).

## Event Details

The following event types are available in our APIs. A table of which event types are available per device type and their triggers can be found on the [Devices](/concepts/devices) page.

### Touch Event

* **Event Type:** `touch`
* **Trigger:** When touched.

Most of our [devices](/concepts/devices) will send a [Touch Event](/concepts/events#touch-event) when touched. This can be useful for identification.

An exception to this is [Counting Touch Sensors](/concepts/devices#counting-touch-sensor), which aggregate each touch into a [Touch Count Event](/concepts/events#touch-count-event) sent every [periodic heartbeat](/concepts/events#periodic-heartbeat). To force a single [Touch Event](/concepts/events#touch-event), hold your finger on the sensor for 3 seconds.

The following snippet shows the event `data` field content of a `touch` event.

```javascript
{
    "touch": {
        "updateTime": "2024-12-06T08:13:15.361624Z"
    }
}
```

<table data-header-hidden><thead><tr><th width="192.7876885297">Field</th><th width="150">Type</th><th width="325.6142100518815">Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Temperature Event

* **Event Type:** `temperature`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat) or when touched.&#x20;
  * **Note:** [2nd Gen Temperature Sensors](https://support.disruptive-technologies.com/hc/en-us/articles/4405644706194-Temperature-Sensor-2nd-Gen-) and [3rd Gen Temperature Sensors](https://support.disruptive-technologies.com/hc/en-us/articles/9389983018268-Temperature-Sensor-3rd-Gen-) will only send temperature on heartbeats, and not when touched.

The following snippet shows the event `data` field content of a `temperature` event.

```javascript
{
    "temperature": {
        "value": 24.9,
        "samples": [
            {
                "value": 24.9,
                "sampleTime": "2024-12-06T08:15:48.223962Z"
            },
            {
                "value": 24.3,
                "sampleTime": "2024-12-06T08:15:18.318751Z"
            },
            {
                "value": 24.4,
                "sampleTime": "2024-12-06T08:14:48.872918Z"
            },
            ...
        ],
        "updateTime": "2024-12-06T08:15:18.318751Z",
        "isBackfilled": false
    }
}
```

<table data-header-hidden><thead><tr><th width="261.51680556917614">Field</th><th width="150">Type</th><th width="296.2868285283635">Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>value</code></td><td><code>float</code></td><td>Temperature value in Celsius.</td></tr><tr><td><code>samples</code></td><td><code>[struct]</code></td><td>A list of temperature samples and timestamps sampled since the previous heartbeat, ordered newest-to-oldest. Its length is equal to the configured samples per heartbeat, which is 1 by default. The first item in the array will always have the same value as the outer <code>value</code> and <code>updateTime</code> fields. </td></tr><tr><td><code>samples[].value</code></td><td><code>float</code></td><td>Sample temperature value in Celsius.</td></tr><tr><td><code>samples[].sampleTime</code></td><td><code>string</code></td><td>Estimated sample timestamp on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format. <a href="/pages/-MdRE44u57QKzcFJaXdw#sample-timestamp-accuracy">Read more</a>. </td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr><tr><td><code>isBackfilled</code></td><td><code>bool</code></td><td>Indicates if the temperature event is backfilled. The backfill feature is supported for <a href="https://support.disruptive-technologies.com/hc/en-us/articles/11432439857948-Temperature-Probe-Sensor">Temperature Probe Sensors</a> and <a href="https://support.disruptive-technologies.com/hc/en-us/articles/9389983018268-Temperature-Sensor">Temperature Sensors with Data Backfill</a>. When this field is <code>true</code>, the temperature value was measured while the sensor was out of reach of an online Cloud Connector, and has been backfilled automatically after getting back online. See the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/12707591585564-How-does-Data-Backfill-work">Backfill page</a> in our Help Center for more information about Data Backfill.</td></tr></tbody></table>

#### Samples per Heartbeat Configuration

Our 2nd and 3rd generation temperature sensors can be configured to sample up to 30 samples per [heartbeat](/concepts/events#periodic-heartbeat). If enabled, these inter-heartbeat samples will be contained in a list found under the `samples` field, accompanied by a timestamp called `sampleTime`. By default, if no configuration is enabled, this list contains only the single temperature value and timestamp found in the temperature event. [Read more](/concepts/usage-examples/2.gen-temperature-sensor).

### Object Present Event

* **Event Type:** `objectPresent`
* **Trigger:** When an object in close proximity appears or disappears.

The following snippet shows the event `data` field content of an `objectPresent` event.

```javascript
{
    "objectPresent": {
        "state": "NOT_PRESENT",
        "updateTime": "2024-12-06T08:37:10.711412Z"
    }
}
```

<table data-header-hidden><thead><tr><th width="172">Field</th><th width="150">Type</th><th width="349.59003831417624">Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>state</code></td><td><code>string</code></td><td>Indicates whether an object is <code>"PRESENT"</code> or <code>"NOT_PRESENT"</code>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Humidity Event

* **Event Type:** `humidity`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat) or when touched.

The following snippet shows the event `data` field content of a `humidity` event.

```javascript
{
    "humidity": {
        "temperature": 22.45,
        "relativeHumidity": 17,
        "samples": [
            {
                "temperature": 22.45,
                "relativeHumidity": 17,
                "sampleTime": "2024-12-06T06:13:46.369000Z"
            },
            {
                "temperature": 22.35,
                "relativeHumidity": 18,
                "sampleTime": "2024-12-06T06:12:45.897000Z"
            }
        ],
        "updateTime": "2024-12-06T06:13:46.369000Z",
        "isBackfilled": false
    }
}
```

<table data-header-hidden><thead><tr><th>Field</th><th width="150">Type</th><th width="320">Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>temperature</code></td><td><code>float</code></td><td>Temperature value in Celsius.</td></tr><tr><td><code>relativeHumidity</code></td><td><code>float</code></td><td>Relative humidity in percent.</td></tr><tr><td><code>samples</code></td><td><code>[struct]</code></td><td>A list of temperature and humidity samples since the previous heartbeat, ordered newest-to-oldest. Its length is equal to the configured samples per heartbeat, which is 1 by default. The first item in the array will always have the same value as the outer <code>temperature</code> and <code>relativeHumidity</code> fields. Only the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/17167989869724-Temperature-and-Humidity-sensor">Temperature &#x26; Humidity Sensor</a> supports configuring the number of samples per heartbeat.</td></tr><tr><td><code>samples[].temperature</code></td><td><code>float</code></td><td>The sampled temperature value in Celsius.</td></tr><tr><td><code>samples[].relativeHumidity</code></td><td><code>float</code></td><td>The sampled relative humidity value in percent.</td></tr><tr><td><code>samples[].sampleTime</code></td><td><code>string</code></td><td>Estimated sample timestamp on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format. <a href="https://disruptive.gitbook.io/docs/concepts/usage-examples/2.gen-temperature-sensor#sample-timestamp-accuracy">Read more</a>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr><tr><td><code>isBackfilled</code></td><td><code>bool</code></td><td>Indicates if the humidity event is backfilled. This feature is supported for the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/17167989869724-Temperature-and-Humidity-sensor">Temperature &#x26; Humidity Sensor</a>. When this field is <code>true</code>, the humidity and temperature value was measured while the sensor was out of reach of an online Cloud Connector, and has been backfilled automatically after getting back online. See the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/12707591585564-How-does-Data-Backfill-work">Backfill page</a> in our Help Center for more information about Data Backfill.</td></tr></tbody></table>

### Object Present Count Event

* **Event Type:** `objectPresentCount`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat).

This event aggregates [Object Present Events](/concepts/events#object-present-event) and sends the total lifetime count every periodic heartbeat.

The following snippet shows the event `data` field content of an `objectPresentCount` event.

```javascript
{
    "objectPresentCount": {
        "total": 4176,
        "updateTime": "2024-12-06T08:23:43.209000Z"
    }
}
```

<table data-header-hidden><thead><tr><th width="189.0447030006124">Field</th><th width="150">Type</th><th width="338.52800308999616">Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>total</code></td><td><code>int</code></td><td>The total number of times the sensor has detected the appearance or disappearance of an object over its lifetime. This can currently not be reset.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Touch Count Event

* **Event Type:** `touchCount`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat).

This event aggregates each [Touch Event](/concepts/events#touch-event) and sends the total lifetime count every periodic heartbeat. To force a single [Touch Event](/concepts/events#touch-event) to be sent, hold your finger on the sensor for 3 seconds.

The following snippet shows the event `data` field content of a `touchCount` event.

```javascript
{
    "touchCount": {
        "total": 469,
        "updateTime": "2024-12-06T08:25:21.604000Z"
    }
}
```

<table data-header-hidden><thead><tr><th width="199.0084518470103">Field</th><th width="150">Type</th><th width="325.49125126709043">Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>total</code></td><td><code>int</code></td><td>The total number of times the sensor has been touched over its lifetime. This can currently not be reset.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Water Present Event

* **Event Type:** `waterPresent`
* **Trigger:** When the sensor detects the appearance or disappearance of water.

The following snippet shows the event `data` field content of a `waterPresent` event.

```javascript
{
    "waterPresent": {
        "state": "PRESENT",
        "updateTime": "2024-12-06T08:43:16.266000Z"
    }
}
```

<table data-header-hidden><thead><tr><th width="191.066583638007">Field</th><th width="152.3681934773962">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>state</code></td><td><code>string</code></td><td>Indicates whether water is <code>"PRESENT"</code> or <code>"NOT_PRESENT"</code>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### CO2 Event

* **Event Type:** `co2`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat).

The following snippet shows the event `data` field content of a `co2` event.

```javascript
{
    "co2": {
        "ppm": 499,
        "updateTime": "2024-12-06T09:02:35.979000Z"
    }
}
```

<table><thead><tr><th>Field</th><th width="162.33333333333331">Type</th><th>Description</th></tr></thead><tbody><tr><td><code>ppm</code></td><td><code>int</code></td><td>CO2 in parts per million.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Pressure Event

* **Event Type:** `pressure`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat).

The following snippet shows the event `data` field content of a `pressure` event.

```javascript
{
    "pressure": {
        "pascal": 98644,
        "updateTime": "2024-12-06T09:02:35.979000Z"
    }
}
```

<table><thead><tr><th>Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td><code>pascal</code></td><td><code>int</code></td><td>Barometric pressure in Pascal.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Motion Event

* **Event Type:** `motion`
* **Trigger:** Every [periodic heartbeat](#periodic-heartbeat).

The following snippet shows the event `data` field content of a `motion` event.

```json
{
    "motion": {
        "state": "MOTION_DETECTED",
        "updateTime": "2024-12-06T11:45:57.454551Z"
    }
}
```

<table><thead><tr><th width="180.67553246294472">Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td><code>state</code></td><td><code>string</code></td><td>Indicates whether <code>MOTION_DETECTED</code> or <code>NO_MOTION_DETECTED</code>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Desk Occupancy Event

* **Event Type:** `deskOccupancy`
* **Trigger:** When the state changes between `OCCUPIED` and `NOT_OCCUPIED`.

The following snippet shows the event `data` field content of a `deskOccupancy` event.

```json
{
    "deskOccupancy": {
        "state": "NOT_OCCUPIED",
        "remarks": [],
        "updateTime": "2024-12-06T14:23:50.728000Z"
    }
}
```

| Field        | Type       | Description                                                                                                                                  |
| ------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `state`      | `string`   | Indicates whether the sensors reports `OCCUPIED` or `NOT_OCCUPIED`.                                                                          |
| `remarks`    | `[string]` | Additional information about the estimated `state` field value. [Read more below](#remarks).                                                 |
| `updateTime` | `string`   | Timestamp of when a Cloud Connector received the event, represented on the [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) format. |

#### Remarks

We use the `remarks` field to relay additional information to the user about the estimated `state` field in the `deskOccupancy` event. If the field is empty, nothing is out of the ordinary.

The `remarks` field can contain none, one, or a combination of the following remarks.

* `INCOMPLETE_DATA`: The model has determined that the occupancy accuracy may be degraded due to insufficient data. If your connection is poor, we recommend using a [Range Extender](https://support.disruptive-technologies.com/hc/en-us/articles/360010450159-Range-Extender) to improve the connection or adding one or more additional Cloud Connectors to extend your coverage.

Remarks are automatically highlighted in DT Studio as shown in the figure below.

<figure><img src="/files/6RGcRRlicT2WIUD0bvP1" alt=""><figcaption></figcaption></figure>

### Contact Event

* **Event Type:** `contact`
* **Trigger:** When the state of a `contact` sensor changes between `OPEN` and `CLOSED`.

The following snippet shows the event `data` field content of a `contact` event.

```json
{
    "contact": {
        "state": "OPEN",
        "updateTime": "2024-12-06T14:23:50.728000Z"
    }
}
```

| Field        | Type     | Description                                                                                                                                  |
| ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `state`      | `string` | Indicates whether the sensors reports `OPEN` or `CLOSED`.                                                                                    |
| `updateTime` | `string` | Timestamp of when a Cloud Connector received the event, represented on the [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) format. |

### Probe Wire Status Event

* **Event Type:** `probeWireStatus`
* **Trigger:** When wires are plugged in or unplugged from a `temperature` probe sensor.

These are the possible state values for the event:

<table><thead><tr><th width="374">State</th><th>Description</th></tr></thead><tbody><tr><td><code>TWO_WIRE</code></td><td>A 2-wire probe has been connected to the sensor.</td></tr><tr><td><code>THREE_WIRE</code></td><td>A 3-wire probe has been connected to the sensor.</td></tr><tr><td><code>FOUR_WIRE</code></td><td>A 4-wire probe has been connected to the sensor.</td></tr><tr><td><code>INVALID_WIRE_CONFIGURATION</code></td><td>Either too few wires are connected to the sensor, or the wires are connected to the wrong terminals. Look at the <a href="https://support.disruptive-technologies.com/hc/en-us/articles/11432439857948-Temperature-Probe-Sensor">Datasheet</a> for more information.</td></tr><tr><td><code>INVALID_COEFFICIENT_CONFIGURATION</code></td><td>The coefficient configuration in Studio does not match the probe that's connected to the sensor. This may happen if a PT100 probe is connected to the sensor, but the coefficients for a PT1000 probe is configured in Studio.</td></tr></tbody></table>

The following snippet shows the event `data` field content of a `probeWireStatus` event.

```json
{
    "probeWireStatus": {
        "state": "THREE_WIRE",
        "updateTime": "2024-12-06T14:23:50.728000Z"
    }
}
```

<table><thead><tr><th width="196">Field</th><th width="129.33333333333331">Type</th><th>Description</th></tr></thead><tbody><tr><td><code>state</code></td><td><code>string</code></td><td>Indicates whether the probe wire state is <code>INVALID_WIRE_CONFIGURATION</code>, <code>INVALID_COEFFICIENT_CONFIGURATION</code>, <code>TWO_WIRE</code>, <code>THREE_WIRE</code> or <code>FOUR_WIRE</code>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Network Status Event

* **Event Type:** `networkStatus`
* **Trigger:** Every [periodic heartbeat](/concepts/events#periodic-heartbeat).

This event describes the current connectivity state of a sensor. It is sent in addition to other events and contains information like the signal strength to the Cloud Connector it propagated through.

#### Proximity to multiple Cloud Connectors

One `networkStatus` event is sent for each Cloud Connector that heard the sensor. As shown in the snippet below, when fetching events through a [stream](https://developer.disruptive-technologies.com/api#/Live%20Events%20Streaming%20API/get_projects__project__devices_stream), [Data Connector](/data-connectors/introduction-to-data-connector), or [event history](https://developer.disruptive-technologies.com/api#/Event%20History/get_projects__project__devices__device__events), each `networkStatus` event contains only a single entry in the `cloudConnectors` field. When [listing a device](https://developer.disruptive-technologies.com/api#/Devices%20%26%20Labels/get_projects__project__devices), these individual `networkStatus` events are merged and all displayed in the `reported` field.

#### Offline detection

Use the [`connectionStatus`](#connection-status-event) event to check whether a sensor is online or offline. This event takes into account the current [Heartbeat Interval](#periodic-heartbeat) of the sensor and only sends out an event with the `OFFLINE` connection status when one heartbeat has been dropped. When the sensor comes back online, a new `connectionStatus` event will be sent out with the `SDS` ([Secure Data Shot](https://www.disruptive-technologies.com/explore/platform-security-how-we-make-sure-sensor-data-is-private-and-secure)) connection status.

The following snippet shows the event `data` field content of a `networkStatus` event.

```javascript
{
    "networkStatus": {
        "signalStrength": 45,
        "rssi": -83,
        "updateTime": "2024-12-06T08:21:21.076013Z",
        "cloudConnectors": [
            {
                "id": "bdkjbo2v0000uk377c4g",
                "signalStrength": 45,
                "rssi": -83
            }
        ],
        "transmissionMode": "LOW_POWER_STANDARD_MODE"
    }
}
```

<table data-header-hidden><thead><tr><th width="375.79298024271867">Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>signalStrength</code></td><td><code>int</code></td><td>The percentage signal strength (0% to 100%) of the strongest Cloud Connector, derived directly from the RSSI value.</td></tr><tr><td><code>rssi</code></td><td><code>int</code></td><td>Raw <a href="https://en.wikipedia.org/wiki/Received_signal_strength_indication">Received Signal Strength Indication</a> as measured by the Cloud Connector with the strongest signal.</td></tr><tr><td><code>cloudConnectors</code></td><td><code>[struct]</code></td><td>The Cloud Connector that forwarded the event.</td></tr><tr><td><code>cloudConnectors[].id</code></td><td><code>string</code></td><td>Unique Cloud Connector identifier.</td></tr><tr><td><code>cloudConnectors[].signalStrength</code></td><td><code>int</code></td><td>The percentage signal strength (0% to 100%) between the sensor and Cloud Connector.</td></tr><tr><td><code>cloudConnectors[].rssi</code></td><td><code>int</code></td><td><a href="https://en.wikipedia.org/wiki/Received_signal_strength_indication">Raw Received Signal Strength Indication</a> between the sensor and Cloud Connector.</td></tr><tr><td><code>transmissionMode</code></td><td><code>string</code></td><td>Indicated whether the sensor is in <code>"LOW_POWER_STANDARD_MODE"</code> or <code>"HIGH_POWER_BOOST_MODE"</code>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Battery Status Event

* **Event Type:** `batteryStatus`
* **Trigger:** Approximately once per day.

The following snippet shows the event `data` field content of a `batteryStatus` event.

```javascript
{
    "batteryStatus": {
        "percentage": 100,
        "updateTime": "2024-12-06T08:21:21.076013Z",
        "state": "OK",
        "estimatedRemainingYears": 14.5
    }
}
```

<table data-header-hidden><thead><tr><th>Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>percentage</code></td><td><code>int</code></td><td>A coarse percentage estimate (0% to 100%) of the remaining battery.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr><tr><td><code>state</code></td><td><code>string</code></td><td>Qualitative status of the battery.</td></tr><tr><td><code>estimatedRemainingYears</code></td><td><code>float</code></td><td>Best-effort estimate of the remaining lifetime of the sensor, in years.</td></tr></tbody></table>

These are the possible status values for the `state` field:

| State      | Meaning                                                                                                                        |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `OK`       | The sensor is expected to continue operating for more than one year.                                                           |
| `LOW`      | The sensor is expected to operate for less than one year. Plan for replacement or review the connectivity of the installation. |
| `CRITICAL` | The sensor may stop working at any time, and should be replaced.                                                               |

### Labels Changed Event

* **Event Type:** `labelsChanged`
* **Trigger:** When a device label is added, modified, or removed.

The following snippet shows the event `data` field content of a `labelsChanged` event.

```javascript
{
    "added": {
        "label-key": "label-value"
    },
    "modified": {
        "label-key": "new-label-value"
    },
    "removed": [
        "remove-key1",
        "remove-key2"
    ]
}
```

<table data-header-hidden><thead><tr><th>Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>added</code></td><td><code>struct</code></td><td>Key- and value pairs of new labels added, both of <code>string</code> type.</td></tr><tr><td><code>modified</code></td><td><code>struct</code></td><td>Key- and value pairs of modified labels, both of <code>string</code> type. The value is the new updated value.</td></tr><tr><td><code>removed</code></td><td><code>[string]</code></td><td>A list of keys of removed labels.</td></tr></tbody></table>

### Connection Status Event

* **Event Type:** `connectionStatus`
* **Trigger:** Changes in connection for a Sensor or Cloud Connector.

This event is sent when a Sensor or Cloud Connector changes its communication protocol. This is the recommended way to determine if a device is online or offline.

These are the possible status values for a device's connection status:

<table><thead><tr><th width="136">Connection</th><th>Description</th></tr></thead><tbody><tr><td><code>SDS</code></td><td>Used for Sensors when they are online. SDS is the <a href="https://www.disruptive-technologies.com/explore/platform-security-how-we-make-sure-sensor-data-is-private-and-secure">Secure Data Shot</a> protocol.</td></tr><tr><td><code>ETHERNET</code></td><td>Used for Cloud Connectors when they are online and communicate to the Cloud through an ethernet cable.</td></tr><tr><td><code>CELLULAR</code></td><td>Used for Cloud Connectors when they are online and communicate to the Cloud through a cellular connection. Note that a cellular connection might have a higher latency than an ethernet connection, so prefer to use an ethernet cable whenever possible.</td></tr><tr><td><code>OFFLINE</code></td><td>Used for both Sensors and Cloud Connectors when they are determined to be offline. Cloud Connectors are determined to be offline 5–15 minutes after they've lost their connection to the Cloud. Sensors are determined to be offline once the Cloud hasn't received a <a href="#periodic-heartbeat">Periodic Heartbeat</a> within the expected time period, depending on the heartbeat interval of the sensor. The current threshold is <code>max(15 min, HeartbeatInterval * 1.5)</code>, but this is subject to change.</td></tr></tbody></table>

For a Cloud Connector, if both `ETHERNET` and `CELLULAR` connections are available, `ETHERNET` is prioritized.

The following snippet shows the event `data` field content of a `connectionStatus` event for a Cloud Connector that is connected through `ETHERNET`.

```javascript
{
    "connectionStatus": {
        "connection": "ETHERNET",
        "available": [
            "CELLULAR",
            "ETHERNET"
        ],
        "updateTime": "2024-12-06T08:21:21.076013Z"
    }
}
```

<table data-header-hidden><thead><tr><th>Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>connection</code></td><td><code>string</code></td><td>Whether the current connection is <code>SDS</code>, <code>ETHERNET</code>, <code>CELLULAR</code>, or <code>OFFLINE</code>.</td></tr><tr><td><code>available</code></td><td><code>[string]</code></td><td>A list of the <code>string</code> types of networks available to the device. For Cloud Connectors, this can contain the values <code>ETHERNET</code>, <code>CELLULAR</code>, or both. For Sensors, it will contain only <code>SDS</code> when its online. This field will be empty when the device's <code>connection</code> is <code>OFFLINE</code>.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Ethernet Status Event

* **Event Type:** `ethernetStatus`
* **Trigger:** Changes in Cloud Connector ethernet connection.

This event is sent when the ethernet status for a Cloud Connector changes, such as when it gets connected to ethernet or when it receives a new IP address. Note that this means that a Cloud Connector has to be connected to ethernet at least once to get the MAC address of the Cloud Connector through an `ethernetStatus` event.&#x20;

The information in this event can be useful for locating a Cloud Connector on the local network or open the necessary ports in a corporate firewall.

The following snippet shows the event `data` field content of an `ethernetStatus` event.

```javascript
{
    "ethernetStatus": {
        "macAddress": "f0:b5:b7:00:0a:08",
        "ipAddress": "10.0.0.1",
        "errors": [
            {"code": 404, "message": "Not found"},
        ],
        "updateTime": "2024-12-06T08:21:21.076013Z"
    }
}
```

<table data-header-hidden><thead><tr><th>Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>macAddress</code></td><td><code>string</code></td><td>The MAC address of the local network interface.</td></tr><tr><td><code>ipAddress</code></td><td><code>string</code></td><td>The IP address of the Cloud Connector on the local network.</td></tr><tr><td><code>errors</code></td><td><code>[struct]</code></td><td>Any errors related to connecting to the local network.</td></tr><tr><td><code>errors[].code</code></td><td><code>int</code></td><td> Error status code.</td></tr><tr><td><code>errors[].message</code></td><td><code>string</code></td><td>A description of the error.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>

### Cellular Status Event

* **Event Type:** `cellularStatus`
* **Trigger:** Changes in Cloud Connector cellular connection strength.

The following snippet shows the event `data` field content of a `cellularStatus` event.

```javascript
{
    "cellularStatus": {
        "signalStrength": 80,
        "errors": [
            {"code": 404, "message": "Not found"},
        ],
        "updateTime": "2024-12-06T08:21:21.076013Z"
    }
}
```

<table data-header-hidden><thead><tr><th width="238.7566864654878">Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>signalStrength</code></td><td><code>int</code></td><td>Cellular reception signal strength (0% to 100%) of the Cloud Connector.</td></tr><tr><td><code>errors</code></td><td><code>[struct]</code></td><td>Any errors related to connecting to the local network.</td></tr><tr><td><code>errors[].code</code></td><td><code>int</code></td><td> Error status code.</td></tr><tr><td><code>errors[].message</code></td><td><code>string</code></td><td>A description of the error.</td></tr><tr><td><code>updateTime</code></td><td><code>string</code></td><td>Timestamp of when a Cloud Connector received the event, represented on the <a href="https://datatracker.ietf.org/doc/html/rfc3339">RFC 3339</a> format.</td></tr></tbody></table>


# Topics

Various overviews, guides, and articles.

* [Temperature Measurement Interval](/concepts/usage-examples/2.gen-temperature-sensor)
* [Motion Sensor Activity Timer](/concepts/usage-examples/motion-sensor-activity-timer)


# Temperature Measurement Interval

A quick showcase on how to interact with the sampling interval of the 2nd and 3rd generation temperature sensors.

## Overview

The 2nd and 3rd Generation Wireless Temperature Sensor both supports configuring a sampling interval that is shorter than the heartbeat interval. This enables the temperature sensor to measure the temperature as often as every 30 seconds, with only a minor impact on battery life. Like our 1st generation sensors, an event is generated every periodic heartbeat, but with the 2nd and 3rd generation hardware, the event can contain up to 30 temperature samples, depending on the configuration.

## Configuring Sampling Interval

The sampling interval can be configured from Studio in the Sensor Configuration section of the device details page.

<figure><img src="/files/RO7MoSz2M4rEOXLZ2ndy" alt=""><figcaption><p>Sensor Configuration in Studio for a 2nd and 3rd Generation Temperature Sensor</p></figcaption></figure>

API support for device configuration is coming later.

## Measurement Timeline

When the sensor is configured to sample more than 1 sample per heartbeat, additional samples are included with every temperature event as a list of timestamps and values. As shown in the [Temperature Event documentation](/concepts/events#temperature-event), the first element in the list is equal to the event timestamp and value, independent of configuration.

If more than 1 sample has been sampled during the period, the list is in decreasing order, meaning that the oldest sample is located at the end. The total length of the list will depend on the configuration.

![How events for the 2nd and 3rd Generation Temperature Sensor are scheduled.](/files/-MdbUhtBdc2LhFxqEC7U)

## Concatenating Historic Events

A [Temperature Event](/concepts/events#temperature-event) has a `samples` field that will always include all the measured values since the previous heartbeat. This makes it easier to create a single sorted list of all the samples when [listing temperature events](https://developer.disruptive-technologies.com/api#/Event%20History/get_projects__project__devices__device__events) through the API.

{% tabs %}
{% tab title="Python API" %}
The following snippet shows how to unpack a list of historic temperature events fetched using our Python API. A full example can be found [here](https://developer.disruptive-technologies.com/api/libraries/python/client/examples/2nd_gen_temperature_sensor.html).

```python
samples = []
for event in events:
    samples += event.data.samples
```

{% endtab %}
{% endtabs %}

## Publishing Emulated Events

Temperature events from the 2nd and 3rd Generation Wireless Temperature Sensor can be generated without access to physical sensors using the [Emulator API](/sensor-emulator).

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click **API Integrations -> Emulator**, then select or create a temperature sensor. When selecting **Emulate sampling between heartbeats**, the feature appears.

![](/files/-MdWqGljUE6uNe_oB_hw)
{% endtab %}

{% tab title="Python API" %}
The [publish\_event()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/emulator.html#publish-event) method is available under the Emulator resource of our Python API. The following snippet shows an overview, and a full implementation example can be found [here](https://developer.disruptive-technologies.com/api/libraries/python/client/examples/2nd_gen_temperature_sensor.html).

```python
dt.Emulator.publish_event(
    device_id="<YOUR_DEVICE_ID>",
    project_id="<YOUR_PROJECT_ID>",
    data=dt.events.Temperature(
        timestamp=datetime.utcnow(),
        celsius=22.3,
        samples=[
            dt.events.TemperatureSample(
                celsius=22.3,
                timestamp=datetime.utcnow(),
            ),
            dt.events.TemperatureSample(
                celsius=22.8,
                timestamp=datetime.utcnow()-timedelta(seconds=30),
            ),
            dt.events.TemperatureSample(
                celsius=23.3,
                timestamp=datetime.utcnow()-timedelta(seconds=60),
            ),
        ],
    )
)
```

{% endtab %}
{% endtabs %}

## Sample Timestamp Accuracy

Normally, the `updateTime` of an event is set to when the Cloud Connector receives the event from the sensor. Since this is not possible for samples between heartbeats, the timestamp of these samples is estimated by DT Cloud.

The implementation of this estimation assumes an ideally distributed interval according to the configuration. For example, if the sensor is configured to 5 samples per heartbeat and it has been exactly 10 minutes since the previous heartbeat, the interval between the timestamps will be exactly 2 minutes.


# Motion Sensor Activity Timer

With the release of our [Motion Sensor](/concepts/devices#motion-sensor), a new mode of operation is introduced, the Activity Timer. Similar to our [Proximity Sensors](/concepts/devices#proximity-sensor), an event is sent at a trigger, here when presence is detected by the sensor. However, the state does not revert until a set timer has expired.

## How it Works

The Wireless Motion Sensor uses Passive Infrared Sensor (PIR) technology to detect the presence of people in a room by measuring changes in infrared light intensity coming from people moving into the sensor's field of view. When the sensor detects presence it will send a `MOTION_DETECTED` event to the cloud and start a pre-set Activity Timer. If the sensor continues to detect the presence of people before the Activity Timer expires, the timer will restart. When the Activity Timer expires, the sensor will send a `NO_MOTION_DETECTED` event to the cloud.

Independent of occupancy events, the sensor will periodically transmit Network Status Events to the cloud containing connectivity information, so that the system can know that the sensor is online and functional.

![](/files/tH8G5Ii4bpOzkU1oxKRP)


# Introduction to Data Connectors

A quick introduction to what Data Connectors are and how they work.

## Overview

Data Connectors are designed to reliably and securely forward live [sensor events](/concepts/events) from DT Cloud to your cloud in real time. This allows you to store sensor data within your systems, and do your processing, analysis, or visualizations as soon as new data becomes available. Data Connectors have a built-in retry mechanism in case your cloud is down for maintenance or is otherwise unavailable, and provides a mechanism to verify the origin and integrity of the sensor events.

Data Connectors are the recommended way to build an integration between DT Cloud and your system whenever possible.

Data Connectors can be summarized in a few steps.

1. A sensor sends a data event to DT Cloud via a Cloud Connector.
2. DT Cloud uses the Data Connector to forward the events to your cloud.
3. Your cloud acknowledges that the event was received.

![](/files/-MTaZT9NJ-VF990L7BFo)

The Data Connector type that is supported today is a webhook. A webhook will forward each sensor event to your server through an HTTPS POST request to the endpoint URL configured on the Data Connector. See the [Receiving Events](/data-connectors/receiving-events) section for more information about how to implement your cloud service.

## Features

Data Connectors have properties that make them particularly well suited for forwarding your sensor data from DT Cloud to an external service with minimal effort.

### At-Least-Once Guarantee

Data Connectors have a retry mechanism that guarantees all events will be pushed at least once for up to 12 hours. All device events received by DT Cloud are put in a dedicated per-Data Connector queue. Events are only removed from this queue either once acknowledged or if the event is older than 12 hours. See the [retry policy](/data-connectors/receiving-events#retry-policy) for more details.

![](/files/-MTaZ_sDXIyx4JmEcKnh)

### Low Latency

Data Connectors give you the lowest possible end-to-end latency from the Sensor to your cloud. DT Cloud will forward each event individually to your cloud as soon as it is received from the sensor. We do not introduce wait states to aggregate events or process events sequentially, keeping latency as low as possible.

![](/files/-MTaZd54SRG3a8BzzuAv)

### Security

All events sent by a Data Connector to an external service are encrypted using TLS with the optional addition of a [signature secret](/data-connectors/advanced-configurations#signing-events) that can be used to verify both the origin and integrity of each event. We recommend using signature secrets with Data Connectors in a production environment. See [Configuring a Data Connector](/data-connectors/advanced-configurations) for more information.

### High Scalability

DT Cloud will never wait for an event delivery to complete before sending the next event. A new event will always be sent instantly to your cloud, running as many requests in parallel as there are events at any given moment.


# Creating a Data Connector

A quick guide on how to create a Data Connector using DT Studio or our APIs.

## Overview

To illustrate the process, we will create a new Data Connector that forwards all events to [webhook.site](https://webhook.site), a free service that fulfills the same role as a server that lets you inspect the contents of each request. If you already have an HTTPS POST endpoint set up, feel free to use that instead.

## Prerequisites

* **Service Account**\
  Creating, updating, and deleting Data Connectors requires that your User or [Service Account](/service-accounts/introduction-to-service-accounts) has been granted the [role](/service-accounts/managing-access-rights#roles-and-permissions) Project Developer or higher.
* **HTTPS Endpoint**\
  Data Connectors sends events as POST requests over HTTPS. In order to receive these events you need to have a service up and running that can accept incoming HTTPS connections.HTTP is not supported. In this guide, we will use [webhook.site](https://webhook.site) as an example.

{% hint style="warning" %}
Please note that Disruptive Technologies and webhook.site are in no way affiliated and that the data sent there is publicly available.
{% endhint %}

## Creating a New Data Connector

Data Connectors can be created using either the DT Studio interface or our REST API. For simplicity, most of the configurations will be left as default in this example. If you're interested in a more custom Data Connector, see the [Configuring a Data Connector](/data-connectors/advanced-configurations) page for more information.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), navigate to your Project. In the left menu, locate **Data Connectors** under **API Integrations** and press **Add Data Connector**. This will open the Data Connector configuration menu.

<figure><img src="/files/h38bf7KVKugbuhr7Ndg1" alt=""><figcaption></figcaption></figure>

Edit the following parameters.

* **Display name**\
  Give the Data Connector some identifiable display name.
* **Endpoint URL**\
  The URL the Data Connector will forward events to. As we use webhook.site in this example, navigate to <https://webhook.site> to generate a new URL on the form `https://webhook.site/<RANDOM_ID>`. Copy this URL, and paste it into the **Endpoint URL** field of the Data Connector config.

![](/files/P7ENZ2kNhSU4XGMaXlxU)

Remember to save your Data Connector configuration at the bottom of the page.&#x20;
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors`

A request body with the following parameters is required.

```javascript
{
    "type": "HTTP_PUSH",
    "displayName": "Example",
    "httpConfig": {
        "url": "<ENDPOINT_URL>"
    }
}
```

A list of all available parameters can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api#/Data%20Connectors/post_projects__project__dataconnectors).

#### Example Usage

Using cURL with a Service Account for authentication, the following example creates a new Data Connector with a given name "Example" and a webhook.site endpoint which can be monitored at `https://webhook.site/<RANDOM_ID>`.

```bash
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"displayName": "Example", "type": "HTTP_PUSH", "httpConfig": {"url": "https://webhook.site/<RANDOM_ID>"}}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a new Data Connector can be created by calling the following resource method.

* [disruptive.DataConnector.create\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#create-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example creates a new Data Connector with a given name "Example" and a webhook.site endpoint.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Create a new HTTP Push Data Connector called "Example".
new_dcon = dt.DataConnector.create_data_connector(
    project_id='<PROJECT_ID>',
    config=dt.DataConnector.HttpPushConfig(
        url='https://webhook.site/<RANDOM_ID>',
    ),
    display_name='Example',
)

# Print the newly created Data Connector.
print(new_dcon)
```

{% endtab %}
{% endtabs %}

## Test Your Data Connector

The simplest way of testing your connection is by forwarding an event in one of the following ways.

* Wait for a sensor in the project to naturally emit an event by its [periodic heartbeat](/concepts/events#periodic-heartbeat).
* Touching a sensor to force an emitted event.
* Use the [Sensor Emulator](/sensor-emulator) to emulate an emitted event.

Once an event has been forwarded by the Data Connector, each event will be visible as individual requests at the generated webhook.site URL.

You can verify that an event has been successfully forwarded by looking at the Data Connector metrics.

{% tabs %}
{% tab title="DT Studio" %}
The Data Connector metrics can be found at the top of the Data Connector page in Studio.

![](/files/-MR5_OtTCI1j52tezwLN)
{% endtab %}

{% tab title="REST API" %}
Send a **GET** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATACONNECTOR>:metrics`

The response contains the metrics for the specified Data Connector over the last 3 hours.

```javascript
{
  "metrics": {
    "successCount": 44,
    "errorCount": 0,
    "latency99p": "0.663s"
  }
}
```

#### Example

Using cURL with a Service Account for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.

```bash
curl -X GET "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:metrics" \
    --user "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), Data Connector metrics can be fetched by calling the following resource method.

* [disruptive.DataConnector.get\_metrics()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#get-metrics)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Fetch the metrics of the specified Data Connector.
metrics = dt.DataConnector.get_metrics(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
)

# Print the content of metrics.
print(metrics)
```

{% endtab %}
{% endtabs %}

If the Error counter is incrementing, it indicates that DT Cloud is unable to forward events to your endpoint. This can be for any of the following reasons:

* You have entered an incorrect Endpoint URL in the Data Connector config.
* An invalid SSL certificate has been configured on the server. See [Server Configuration](/data-connectors/receiving-events#server-configuration) for more details.
* If you're forwarding events to a cloud function, make sure the cloud function is publicly available and does not require authentication. Otherwise, DT Cloud will receive 401 or 403 error codes from your endpoint.
* Check your error logs for any other issues.


# Configuring a Data Connector

Data Connectors can be configured for different use-cases.

## Preliminaries

Before diving into how to configure a Data Connector, there are a few things worth keeping in mind when working with them.

* **Multiple Projects**\
  Data Connectors are created and configured on a per-project basis. If you want multiple projects to send data to the same Endpoint URL, you have to create one Data Connector per project.
* **Multiple Data Connectors**\
  All Data Connectors in a project will receive events from all the devices in that project. It is recommended to filter on event types when possible.
* **Access Control**\
  To create, update, or delete a Data Connector, your DT Studio User or [Service Account](/service-accounts/introduction-to-service-accounts) must have the [role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher.

## Enable and Disable

A Data Connector can be enabled or disabled at any time. A Data Connector that is disabled will not forward events to its Endpoint URL until it is re-enabled. Any events that were generated while the Data Connector was disabled will not be forwarded once the Data Connector is re-enabled.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click on **API Integrations -> Data Connectors**. Select the desired Data Connector, then toggle the **Data Connector enabled** switch. Remember to click **Update Data Connector**.

![](/files/-Me_A5VF6vayWHvGFWot)
{% endtab %}

{% tab title="REST API" %}
Send a **PATCH** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>`

A request body with the following parameter should be included.

```javascript
{
    "status": "ACTIVE"
}
```

The `status` field of a Data Connector can be set to either `"ACTIVE"` or `"USER_DISABLED"`. If the Data Connector gets automatically disabled by DT Cloud, the status will be set to `SYSTEM_DISABLED`. See the [Auto-Disabling of Data Connectors](/data-connectors/advanced-configurations#auto-disabling-of-data-connectors) section for more information.

#### Example Usage

Using cURL with a Service Account for authentication, the following example disables the speficied Data Connector by setting `status` to `"USER_DISABLED"`.

```bash
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"status": "USER_DISABLED"}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Data Connector can be updated by calling the following resource method.

* [disruptive.DataConnector.update\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#update-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example disables the specified Data Connector by setting `status` to `"USER_DISABLED"`.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Disabled Data Connector by updating the status.
dcon = dt.DataConnector.update_data_connector(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
    status='USER_DISABLED',
)

# Print the updated Data Connector.
print(dcon)
```

The `status` field of a Data Connector can be set to either `"ACTIVE"` or `"USER_DISABLED"`. If the Data Connector gets automatically disabled by DT Cloud, the status will be set to `SYSTEM_DISABLED`. See the [Auto-Disabling of Data Connectors](/data-connectors/advanced-configurations#auto-disabling-of-data-connectors) section for more information.
{% endtab %}
{% endtabs %}

{% hint style="danger" %}
Disabled Data Connectors have no at-least-once guarantee.

When a Data Connector is disabled, undelivered events and events generated after being disabled will not be sent. Re-enabling the Data Connector will **not** backfill data from the period it was disabled and must be fetched programmatically using our REST API.
{% endhint %}

#### Auto-Disabling of Data Connectors

A Data Connector will be automatically disabled by DT Cloud if it has failed consistently for an extended period of time (currently 5 days) without any success. A push-attempt from a Data Connector is considered a failure if the receiver responded with a non-2xx status code, or didn’t respond at all. A Data Connector that is disabled by DT Cloud will get the status `SYSTEM_DISABLED`.

## Filtering on Event Types

By default, all [event types](/concepts/events#types) are forwarded by a Data Connector. If you want to avoid unnecessary traffic on your endpoint, events can be filtered by type.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click on **API Integrations -> Data Connectors**. Select the desired Data Connector, then toggle the **Forward All Events** followed by a selection. Remember to click **Update Data Connector**.

![](/files/-MRKsR153QutvaKaSV7E)
{% endtab %}

{% tab title="REST API" %}
Send a **PATCH** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>`

A request body with the following parameter should be included.

```javascript
{
    "events": ["touch", "temperature", ...]
}
```

The events array takes one- or several [event types](/concepts/events#types).

#### Example Usage

Using cURL with a Service Account for authentication, the following example sets a filter so that only temperature- and humidity events are forwarded.

```bash
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"events": ["temperature", "humidity"]}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Data Connector can be updated by calling the following resource method.

* [disruptive.DataConnector.update\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#update-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example sets a filter so that only temperature- and humidity events are forwarded.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Update the Data Connector to only forward specified events.
dcon = dt.DataConnector.update_data_connector(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
    event_types=[
        dt.events.TEMPERATURE,
        dt.events.HUMIDITY,
    ],
)

# Print the updated Data Connector.
print(dcon)
```

{% endtab %}
{% endtabs %}

## Including Labels

A Data Connector can optionally forward device labels along with each event. By default, no device labels will be included, but you can specify a list of label keys that should be included in each request. See the [Request Contents](/data-connectors/receiving-events#request-contents) section for more details about how these device labels are included in the requests.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click on **API Integrations -> Data Connectors**. Select the desired Data Connector, then append the label keys to the list. Remember to click **Update Data Connector**.

![](/files/-MRKwFGIes8Bkm0bEdlp)
{% endtab %}

{% tab title="REST API" %}
Send a **PATCH** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>`

A request body with the following parameter should be included.

```javascript
{
    "labels": ["name", "floor", ...]
}
```

#### Example Usage

Using cURL with a Service Account for authentication, the following example configures the specified Data Connector to include device labels called `"floor"` and `"customer-ID"`.

```bash
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"labels": ["name", "floor"]}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Data Connector can be updated by calling the following resource method.

* [disruptive.DataConnector.update\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#update-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example configured a Data Connector to include device labels called `"name"` and `"floor"`.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Update the Data Connector to include device labels.
dcon = dt.DataConnector.update_data_connector(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
    labels=['name', 'floor'],
)

# Print the updated Data Connector.
print(dcon)
```

{% endtab %}
{% endtabs %}

#### DT Studio Specific Labels

DT Studio uses labels with the keys `name` and `description` as the display name and the description of a device. It is recommended that an integration uses these labels in the same way when displaying or updating a device.

## Adding HTTP Headers

Custom HTTP headers can be included with each event forwarded by a Data Connector.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click on **API Integrations -> Data Connectors**. Select the desired Data Connector, then add a key- and value pairs for your custom headers. Remember to click **Update Data Connector**.

![](/files/-MRKxLfebWy6oY-fVROL)
{% endtab %}

{% tab title="REST API" %}
Send a **PATCH** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>`

A request body with the following parameter should be included.

```javascript
{
    "httpConfig": {
        "headers": {
            "name-1": "value-1",
            "name-2": "value-2",
            ...
        }
    }
}
```

#### Example Usage

Using cURL with a Service Account for authentication, the following example adds the header `"my-header"` with value `"my-value"` to every event forwarded by a Data Connector.

```bash
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"httpConfig": {"headers": {"my-header": "my-value"}}}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Data Connector can be updated by calling the following resource method.

* [disruptive.DataConnector.update\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#update-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example adds the header `"my-header"` with value `"my-value"` to every forwarded event.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Update the Data Connector to include custom HTTP headers.
dcon = dt.DataConnector.update_data_connector(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
    config=dt.DataConnector.HttpPushConfig(
        headers={
           'my-header': 'my-value',
        },
    ),
)

# Print the updated Data Connector.
print(dcon)
```

{% endtab %}
{% endtabs %}

## Signing Events

For increased security in a production integration, we recommend configuring a Signature Secret which can be used to[ verify the signed events](/data-connectors/receiving-events#verifying-signed-events) on the receiving end.

* The HTTP header of the webhook request includes a [JSON Web Token (JWT)](https://jwt.io/) that is signed with the Signature Secret.
* The JWT payload contains the checksum of the request body.

This allows you to validate the origin and the content integrity of the received request for a more secure integration. The process is two-fold and must be configured for both the Data Connector and the receiving [endpoint](/data-connectors/receiving-events#verifying-signed-events).

#### Signing Events Sent by the Data Connector

Adding a signature secret to your Data Connector will automatically sign each forwarded event.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click on **API Integrations -> Data Connectors**. Select the desired Data Connector, then add a strong and unique signature secret. Remember to click **Update Data Connector**.

![](/files/-MT5kF_zzQkxMbZu3sME)
{% endtab %}

{% tab title="REST API" %}
Send a **PATCH** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>`

A request body with the following parameter should be included.

```javascript
{
    "httpConfig": {
        "signatureSecret": "some-good-secret"
    }
}
```

#### Example Usage

Using cURL with a Service Account for authentication, the following example adds a signature secret to the specified Data Connector.

```bash
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"httpConfig": {"signatureSecret": "some-good-secret"}}
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Data Connector can be updated by calling the following resource method.

* [disruptive.DataConnector.update\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#update-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example adds a signature secret to the specified Data Connector.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Update the Data Connector with a signature secret.
dcon = dt.DataConnector.update_data_connector(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
    config=dt.DataConnector.HttpPushConfig(
        signature_secret='<YOUR_SECURE_SECRET>',
    ),
)

# Print the updated Data Connector.
print(dcon)
```

{% endtab %}
{% endtabs %}

#### Verifying Signed Events

To handle incoming events that have been signed by a signature secret, see [Verifying Signed Events](/data-connectors/receiving-events#verifying-signed-events).

## Synchronize

When synchronizing a Data Connector, the most recent event of each event type of each device currently in the project will be published to the Data Connector. This can be useful when a new Data Connector has been set up, and you need to get an up-to-date snapshot of each device in the project.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), click on **API Integrations -> Data Connectors**. Select the desired Data Connector, then click **Synchronize Data Connector** towards the bottom of the page.

![](/files/-Me_gNinQZQF5YzDF4Xm)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR>:sync`

#### Example Usage

Using cURL with a Service Account for authentication, the following example syncs the specified Data Connector.

```bash
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:sync" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Data Connector can be synced by calling the following resource method.

* [disruptive.DataConnector.sync\_data\_connector()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/data_connector.html#sync-data-connector)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example syncs the specified Data Connector.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Synchronize the specified Data Connector.
dt.DataConnector.sync_data_connector(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
)
```

{% endtab %}
{% endtabs %}


# Receiving Events

A few things to consider when receiving events forwarded by a Data Connector.

## Request Contents

Events are delivered to the receiving endpoint as HTTPS POST requests.

Both the headers and body of the incoming request contain information of interest that can be extracted and, depending on the [configuration](/data-connectors/advanced-configurations), used to verify the content and origin of the request.

{% hint style="info" %}
**Exploring the request contents**

You can use [webhook.site](https://webhook.site) to explore the contents of the POST requests. This site will generate a URL that you can point a Data Connector to, and it will display all the relevant details about each request.

Note that using this service will make the events publicly available through the URL generated by that site. We recommend exploring this with emulators in a separate project in DT Studio.
{% endhint %}

### Header

If a [signature secret](/data-connectors/advanced-configurations#signing-events) is set in the Data Connector configuration, the following header will be included.

* **X-Dt-Signature**\
  Includes a [JSON Web Token](https://jwt.io/) (JWT). Once decoded using the signature secret, it contains all the necessary information to verify the request content and origin.

Note that depending on the framework used to receive the Data Connector events, the header name casing may differ. Some services will force header names to be lower-cased, like `x-dt-signature`.

Read more about validating the request signature in the [Verifying Signed Events](#verifying-signed-events) section below.

### Body

The request body contains three fields, `event`, `labels`, and `metadata`. The following snippet shows an example request body of a `touch` event for a `humidity` sensor forwarded by a Data Connector.

```javascript
{
    "event": {
        "eventId": "<EVENT_ID>",
        "targetName": "projects/<PROJECT_ID>/devices/<DEVICE_ID>",
        "eventType": "touch",
        "data": {
            "touch": {
                "updateTime": "2021-05-28T08:34:06.225872Z"
            }
        },
        "timestamp": "2021-05-28T08:34:06.225872Z"
    },
    "labels": {
        "room-number": "99"
    },
    "metadata": {
        "deviceId": "<DEVICE_ID>",
        "projectId": "<PROJECT_ID>",
        "deviceType": "humidity",
        "productNumber": "102081"
    }
}
```

<table data-header-hidden><thead><tr><th width="194.4794101688501">Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>Field</td><td>Type</td><td>Description</td></tr><tr><td><code>event</code></td><td><code>struct</code></td><td>Contains event data. See the <a href="/pages/-MR5VndbmI4ns9Ey_BE2#structure">Event</a> documentation where the structure is explained in detail for each event type.</td></tr><tr><td><code>labels</code></td><td><code>struct</code></td><td>Device label key- and value pairs included by the Data Connector. See the <a href="/pages/-MS7_KNvJyF_sxlY1fBW#including-labels">Advanced Configuration</a> page for details about including labels.</td></tr><tr><td><code>metadata</code></td><td><code>struct</code></td><td>Contains metadata about the device that is the source of the event. See the section below for more details.</td></tr></tbody></table>

#### Event Metadata

Each event has a `metadata` field that includes details about the device that is the source of the event. The event metadata has the following structure.

| Field           | Type     | Description                                                                               |
| --------------- | -------- | ----------------------------------------------------------------------------------------- |
| `deviceId`      | `string` | The identifier of the device that published the event.                                    |
| `projectId`     | `string` | The identifier of the project the device is in.                                           |
| `deviceType`    | `string` | The [device type](/concepts/devices#device-types) that published the event.               |
| `productNumber` | `string` | The [product number](/concepts/devices#structure) of the device that published the event. |

{% hint style="info" %}
**Note**

The structure of the `metadata` field might change in the future if event types are added that are not published by devices. Make sure to first check `event.eventType` to make sure it is a known device event before processing the `metadata` field.

See the code sample below for an example of how to do this.
{% endhint %}

The event metadata makes it possible to check which [device type](/concepts/devices#device-types) has published the event, even for event types like `touch` or `networkStatus` which are published by many types of devices. This makes it possible to add new devices to a database without having to first look up the device using the REST API.

The metadata also provides a more convenient way to get the `deviceId` and `projectId` of the device that published the event, without having to parse the `event.targetName` field.

## Implementing Your Endpoint

In order to receive events from a Data Connector, your server needs to be set up to do the following:

* Start listening for incoming requests on the URL specified in the Data Connector's Endpoint URL.
* Read the HTTP headers and body, and process the event.
* Reply with a **2XX** status code in a timely manner. Any response code outside the 2xx range will be considered a failed delivery and will be [retried](#retry-policy).

If your endpoint fails to reply with status codes in the 2xx range consistently for an extended time period, the Data Connector will eventually be [automatically disabled](/data-connectors/advanced-configurations#auto-disabled-data-connectors).

This server can be written in any programming language. Example implementations in a selection of languages can be found on the [Example Integrations](/data-connectors/example-integrations) page, as well as in the [Verifying Signed Events](#verifying-signed-events) section on this page.

Regardless of which language is used to implement the server, there are a few things to keep in mind:

#### Server Configuration

The server needs to be set up to listen for HTTPS POST requests on the URL specified in the Data Connector's Endpoint URL. This endpoint needs to be publicly available and not require any authorization. You can verify that the event originates from DT by following the steps in the Verifying Signed Events section.

The server also needs to be configured with a valid SSL certificate that is issued from one of the root certificates from [Mozilla's CA Certificate Program](https://wiki.mozilla.org/CA). This will be the case for most certificates (e.g. certificates issued by Let's Encrypt, DigiCert, GlobalSign, etc). Self-signed certificates are not supported.

{% hint style="info" %}
**SSL Certificate Verification**

You can verify that you have a valid SSL cert by running `curl -v {YOUR_ENDPOINT}` in your terminal and look for the string "SSL certificate verify ok".

To verify that your SSL cert is issued by one of the root certs in Mozilla's CA Certificate Program, you can spin up an Alpine docker container locally, and check your SSL cert from that container. Alpine uses the root certs in Mozilla's CA Certificate Program to validate SSL certificates. Run the following commands in your terminal (assuming Docker is installed):

1. `docker run --rm -ti alpine:latest sh`
2. `apk --update add ca-certificates curl`
3. `curl -v {YOUR_ENDPOINT}`
4. Make sure the "SSL certificate verify ok" string is present in the output
5. Use ctrl+d to quit the container

To help diagnose any SSL issues, you can use the [SSL Server Test](https://www.ssllabs.com/ssltest/analyze.html) from SSL Labs.
{% endhint %}

#### Handle Incoming Event

Each event will be delivered to your endpoint separately as individual requests. When processing a request, you should do the following steps:

1. (Recommended) Verify the integrity of the event, and that it originates from DT (see [Verify Signed Events](#verifying-signed-events) below).
2. Do something with the event. You might for example put in on a separate queue for later processing (Google Pubsub, Amazon SQS, Azure ServiceBus, etc), write it to a database, or do some other processing.
3. Respond with a status code in the 2xx range once you've accepted/processed the event.

These steps should all be done in a timely manner, and respond within 10 seconds. If your endpoint takes more than 10 seconds to respond, you run the risk that it will time out and be retried according to the [Retry Policy](#retry-policy).

If you think there's a possibility that your processing might take more than 10 seconds, prefer to write the event to a separate queue or a database and do the processing at a later point in time.

## Handling Duplicates

Every event received by DT Cloud is put in a [dedicated, per-Data Connector queue](/data-connectors/introduction-to-data-connector#at-least-once-guarantee). Messages are removed from this queue once acknowledged, or if the message is older than 12 hours.

A side effect of this delivery guarantee is that, under certain conditions, **you may receive duplicates** of the same event. While rare, deduplication should be performed on the receiving end by checking event IDs.

{% hint style="success" %}
**Best Practice**

Use the included **eventId** field to check for duplicated events.
{% endhint %}

## Retry policy

Any time a Data Connector does not receive a successful response (**HTTP status code 2xx**), the event will be retried. If an event has not been successfully acknowledged after 12 hours, it will be discarded.&#x20;

The retry interval is calculated as an exponential backoff policy, given by

$$
t\_0\cdot2^{n-1},
$$

where $$t0$$ is the initial interval of 8 seconds and $$n$$ the attempt number. The interval will not exceed 1 hour. For very slow endpoints, the minimum retry interval will be $$4x$$ the response time.

The following table shows the retry interval after a given number of delivery attempts:

<table data-header-hidden><thead><tr><th width="332">Attempt</th><th>Retry Interval [s]</th></tr></thead><tbody><tr><td>Attempt</td><td>Retry Interval [s]</td></tr><tr><td>1</td><td>8</td></tr><tr><td>2</td><td>16</td></tr><tr><td>3</td><td>32</td></tr><tr><td>...</td><td>...</td></tr><tr><td>9</td><td>2048</td></tr><tr><td>10</td><td>3600</td></tr><tr><td>11</td><td>3600</td></tr></tbody></table>

## Verifying Signed Events

When using a Signature Secret, the [X-Dt-Signature](/data-connectors/receiving-events#header) header is included and contains a [JWT](https://jwt.io/), signed by the Signature Secret. Inside, a checksum of the request body can be found and used to check for tampering.&#x20;

The following steps sum up the process of verifying the received request at the receiving endpoint.

1. Extract the signed JWT from the HTTP header **X-Dt-Signature** of the received request.
2. Verify the JWT's signature with the **signature secret.**
3. Calculate a **SHA256** checksum over the entire request body.
4. Compare the body checksum with the `checksum_sha256` field contained in the JWT (there's a `checksum` field as well that uses SHA1 which is less secure than SHA256, and is kept only for backward compatibility).
5. If these checksums are identical, you can be certain that the event has not been tampered with and originated from your Data Connector.

The following snippet from our [Google Cloud Function example integration](/data-connectors/example-integrations/google-cloud-functions) implements this verification process.

{% tabs %}
{% tab title="Python 3.11" %}

```python
# This Python script is built on Flask, docs are available here:
# https://flask.palletsprojects.com/en/2.0.x/quickstart/

import os
import hashlib
import jwt                        # pip install pyjwt==2.7.0
from flask import Flask, request  # pip install Flask==2.3.2

app = Flask(__name__)

# Read environment variable.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')


@app.route('/', methods=["POST"])
def data_connector_endpoint():
    # Extract the body as a bytestring and the signed JWT.
    # We'll use these values to verify the request.
    payload = request.get_data()
    token = request.headers['x-dt-signature']

    # Verify request origin and content integrity.
    if not verify_request(payload, token):
        return ('Could not verify request.', 400)

    # We now know the request came from DT Cloud, and the integrity
    # of the body has been verify. We can now handle the event safely.
    handle_event(request.get_json())

    # Respond with a 200 status code to ack the event. Any states codes
    # that are outside the 2xx range will nack the event, meaning it will
    # be retried later.
    return ('OK', 200)


def verify_request(body, token):
    """
    Verifies that the request originated from DT, and that the body
    hasn't been modified since it was sent. This is done by verifying
    that the checksum field of the JWT token matches the checksum of the
    request body, and that the JWT is signed with the signature secret.
    """

    # Decode the JWT, and verify that it was signed using the
    # signature secret. Also verifies that the algorithm used was HS256.
    try:
        payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
    except Exception as err:
        print(err)
        return False

    # Verify the request body checksum.
    m = hashlib.sha256()
    m.update(body)
    checksum = m.digest().hex()
    if payload["checksum_sha256"] != checksum:
        print('Checksum Mismatch')
        return False

    return True


def handle_event(body):
    """
    Processes the event itself. For this example, we will just
    decode a touch event, and print out the timestamp, device ID,
    and the device type.
    """
    # First, check if the event type is one of the event
    # types we're expecting.
    # As an example, we'll check for touch events here.
    if body['event']['eventType'] == 'touch':
        # Now that we know this is a device event, we can
        # check for the device type and device identifier
        # in the event metadata.
        device_type = body['metadata']['deviceType']
        device_id = body['metadata']['deviceId']
        timestamp = body['event']['data']['touch']['updateTime']

        print("Got touch event at {} from {} sensor with id {}".format(
            timestamp,
            device_type,
            device_id,
        ))
```

{% endtab %}

{% tab title="Python API" %}

```python
# This Python is built on Flask, docs are available here:
# https://flask.palletsprojects.com/en/2.0.x/quickstart/

import os

# pip install dtintegrations==0.5.1
from dtintegrations import data_connector, provider
from flask import Flask, request  # pip install Flask==2.3.2

app = Flask(__name__)

# Read environment variable.
DT_SIGNATURE_SECRET = os.getenv('DT_SIGNATURE_SECRET')


@app.route('/', methods=['POST'])
def dataconnector_endpoint():
    # Validate and decode the incoming request.
    payload = data_connector.HttpPush.from_provider(
        request,
        provider=provider.FLASK,
        secret=DT_SIGNATURE_SECRET,
    )

    # Print the event data.
    handle_event(payload)

    # If all is well, return 200 response.
    return ('OK', 200)


def handle_event(payload):
    """
    Processes the event itself. For this example, we will just
    decode a touch event, and print out the timestamp, device ID,
    and the device type.
    """
    # First, check if the event type is one of the event
    # types we're expecting.
    # As an example, we'll check for touch events here.
    if payload.event.event_type == 'touch':
        # Now that we know this is a device event, we can
        # check for the device type and device identifier
        # in the event metadata.
        metadata = payload.get_device_metadata()
        if metadata is None:
            device_type = metadata.device_type
            device_id = metadata.device_id
            timestamp = payload.event.data.timestamp

            print("Got touch event at {} from {} sensor with id {}".format(
                timestamp,
                device_type,
                device_id,
            ))
```

{% endtab %}

{% tab title="Node.js 16" %}

```javascript
const crypto = require('crypto')
const express = require('express')  // npm install express@4.17.2
const jwt = require('jsonwebtoken') // npm install jsonwebtoken@8.5.1

// Read environment variable
const signatureSecret = process.env.DT_SIGNATURE_SECRET

// dataConnectorEndpoint receives, validates, and returns a response 
// for the forwarded event.
const dataConnectorEndpoint = (req, res) => {
    
    // Validate request origin and content integrity.
    let token = req.headers['x-dt-signature']
    if (verifyRequest(JSON.stringify(req.body), token) === false) {
        res.sendStatus(400)
        return
    }
    
    // We now know the request came from DT Cloud, and the integrity
    // of the body has been verify. We can now handle the event safely.
    handleEvent(req.body)
    
    // Respond with a 200 status code to ack the event. Any states codes
    // that are outside the 2xx range will nack the event, meaning it will
    // be retried later.
    res.sendStatus(200);
};

// Verifies that the request originated from DT, and that the body
// hasn't been modified since it was sent. This is done by verifying
// that the checksum field of the JWT token matches the checksum of the
// request body, and that the JWT is signed with the signature secret.
const verifyRequest = (payload, token) => {
    // Decode the JWT, and verify that it was signed using the 
    // signature secret. Also verifies that the algorithm used was HS256.
    let decoded
    try {
        decoded = jwt.verify(token, signatureSecret, { algorithms: "HS256" })
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    const hash = crypto.createHash("sha256")
    const checksum = hash.update(payload).digest("hex")
    if (checksum !== decoded.checksum_sha256) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

// handleEvent processes the event itself. For this example,
// we will just decode a touch event, and print out the timestamp,
// device ID, and the device type.
const handleEvent = (payload) => {
    // First, check if the event type is one of the event
    // types we're expecting. 
    // As an example, we'll check for touch events here.
    switch (payload.event.eventType) {
        case 'touch':
            // Now that we know this is a device event, we can 
            // check for the device type and device identifier 
            // in the event metadata.
            const deviceType = payload.metadata.deviceType
            const deviceId = payload.metadata.deviceId
            const timestamp = payload.event.data.touch.updateTime
    
            console.log(`Received touch event at ${timestamp} from ${deviceType} sensor with id ${deviceId}`)
            break
        default:
            break
    }
}

// Sets up a bare-bones server that listens on port 8080, and routes
// all requests to the path "/" to the `dataConnectorEndpoint` function
const app = express()
app.use(express.json())
app.listen(8080)
app.post('/', dataConnectorEndpoint)

```

{% endtab %}

{% tab title="Go 1.20" %}

```go
package main

import (
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"

    // go get github.com/golang-jwt/jwt/v5@v5.0.0
    jwt "github.com/golang-jwt/jwt/v5"
)

// Read environment variable
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")

// DataConnectorEndpoint receives, validates, and returns a response
// for the forwarded event.
func DataConnectorEndpoint(w http.ResponseWriter, r *http.Request) {

    // Extract the body and the signed JWT.
    // We'll use these values to verify the request.
    tokenString := r.Header.Get("x-dt-signature")
    bodyBytes, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Println(err)
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // Validate request origin and content integrity.
    if err := verifyRequest(bodyBytes, tokenString); err != nil {
        fmt.Println(err)
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // We now know the request came from DT Cloud, and the integrity
    // of the body has been verify. We can now handle the event safely.
    if err := handleEvent(bodyBytes); err != nil {
        fmt.Println(err)
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // Respond with a 200 status code to ack the event. Any states codes
    // that are outside the 2xx range will nack the event, meaning it
    // will be retried later.
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("OK"))
}

// verifyRequest verifies that the request originated from DT, and that
// the body hasn't been modified since it was sent. This is done by
// verifying that the checksum field of the JWT token matches the checksum
// of the request body, and that the JWT is signed with the signature secret.
func verifyRequest(bodyBytes []byte, tokenString string) error {
    // Decode the JWT, and verify that it was signed using the
    // signature secret. Also verifies the algorithm used to sign the JWT.
    token, err := jwt.Parse(
        tokenString,
        func(token *jwt.Token) (interface{}, error) {
            // Return out signature secret to verify that it was used to
            // sign the JWT.
            return []byte(signatureSecret), nil
        },
        jwt.WithValidMethods([]string{"HS256"}),
    )
    if err != nil {
        return err
    }

    // Verify the request body checksum.
    sha256Bytes := sha256.Sum256(bodyBytes)
    sha256String := hex.EncodeToString(sha256Bytes[:])
    claims := token.Claims.(jwt.MapClaims)
    if sha256String != claims["checksum_sha256"] {
        return fmt.Errorf("Checksum mismatch.")
    }

    return nil
}

// handleEvent processes the event itself. For this example,
// we will just decode a touch event, and print out the timestamp,
// device ID, and the device type.
func handleEvent(payload []byte) error {
    // The structure of the events we'll received from a Data Connector.
    type Event struct {
        Event struct {
            EventId   string          `json:"eventId"`
            EventType string          `json:"eventType"`
            Data      json.RawMessage `json:"data"`
            Timestamp string          `json:"timestamp"`
        } `json:"event"`
        Labels   map[string]string `json:"labels"`
        Metadata map[string]string `json:"metadata"`
    }

    // The structure of the `Event.Data` field for a touch event.
    // We'll be using touch events for this example.
    type TouchData struct {
        Touch struct {
            Timestamp string `json:"updateTime"`
        } `json:"touch"`
    }

    // Decode the event
    var event Event
    if err := json.Unmarshal(payload, &event); err != nil {
        return err
    }

    // First, check if the event type is one of the event
    // types we're expecting.
    // As an example, we'll check for touch events here.
    switch event.Event.EventType {
    case "touch":
        // Now that we know this is a touch event, we can decode
        // the `Event.Data` field.
        var touchData TouchData
        err := json.Unmarshal(event.Event.Data, &touchData)
        if err != nil {
            return err
        }

        // Also, since we now know this is a device event, we can
        // check for the device type and device identifier
        // in the event metadata.
        deviceType := event.Metadata["deviceType"]
        deviceId := event.Metadata["deviceId"]
        timestamp := touchData.Touch.Timestamp

        fmt.Printf("Received touch event at %s from %s sensor with id %s\n",
            timestamp,
            deviceType,
            deviceId,
        )
    }

    return nil
}

func main() {
    // Sets up a bare-bones server that listens on port 8080, and
    // routes all requests to the path "/" to the
    // `DataConnectorEndpoint` function.

    http.HandleFunc("/", DataConnectorEndpoint)

    fmt.Println("Started listening on localhost:8080 ...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Printf("Closed with error: %v\n", err)
    } else {
        fmt.Println("Closed server successfully")
    }
}
```

{% endtab %}
{% endtabs %}


# Best Practices

Some Best Practices to keep in mind when implementing your endpoint.

### **Only subscribe to the events that are needed**

You should use the event filter of the Data Connector to only forward the event types you need. This will reduce the amount of work your server needs to do. See the [Configuring a Data Connector](https://disruptive.gitbook.io/docs/data-connectors/advanced-configurations#filtering-on-event-types) page for more information.

### **Respond within 10 seconds**

Your endpoint should complete its work and respond with a status code in the 2xx range within 10 seconds. If your endpoint takes much longer than this, it will be considered a failed delivery and will be retried according to the [Retry Policy](https://disruptive.gitbook.io/docs/data-connectors/receiving-events#retry-policy)

If you think there's a possibility that your processing might take more than 10 seconds, prefer to write the event to a separate queue or a database and do the processing at a later point in time.

### **Use the `eventId` for deduplication**

Your endpoint should be prepared to handle duplicate deliveries. While rare, in some scenarios, DT Cloud will forward the same event two or more times through a Data Connector. This can happen in scenarios such as network errors, timeouts, or other things where DT Cloud is not certain that the event has been correctly forwarded to your endpoint. This behavior is what provides an at-least-once guarantee for Data Connectors.

To make sure your endpoint is ready to handle these duplicate events, you should check the `event.eventId` field and check if it's an event you've already processed.

If a duplicate event is sent, it will often be sent within a few seconds of the original event. However, we can not provide any guarantees about the timing of these duplicates.

### Use timestamp as a version for events

Similarly to the point above about handling duplicate events, there's also a possibility that events might be delivered out-of-order. If you're keeping track of the most recent events for each device, it is recommended that you use the timestamp of the event as a version. Only overwrite the event for a device if the timestamp is greater than the previous event you had stored.

### **Use a valid SSL certificate issued by a well-known CA**

DT Cloud will always verify the SSL certificates of a URL before delivering events to it. The SSL certificate needs to be issued by one of the Certificate Authorities in [Mozilla's CA Certificate Program](https://wiki.mozilla.org/CA). See the [Server Configuration](/data-connectors/receiving-events#server-configuration) section for more information.

### **Check the event type before processing the event**

Before you start decoding the `event.data` field of an event, you should make sure that the event you're processing is one of the expected event types by checking the `event.eventType` field. You should do this even if you're only subscribing to one event type. This way, your integration will not break or throw unexpected errors if you ever start subscribing to different event types, or if a new event type is added to the API.

### **Disable Data Connector while not in use**

If you don't need a Data Connector for some time (for example if you're done with some testing), it's a good practice to disable it and then re-enable it when you need it again. This is because DT Cloud will automatically disable a Data Connector if it has had errors continuously for an extended time period (it will be set to the state `SYSTEM_DISABLED`). Disabling it yourself instead will give you more control over when this disabling happens, so you will not be caught by surprise that it's been disabled.

See more in the [Auto-Disabling of Data Connectors](/data-connectors/advanced-configurations#auto-disabling-of-data-connectors) section for more information.


# Example Integrations

A few example integrations using Data Connectors with various popular cloud services.

## Overview

These example integrations aim to point you in the right direction for using Data Connectors to integrate with popular cloud services. To get you up and running quickly, necessary concepts are briefly explained, and example code is provided in a few languages.

* [Heroku](/data-connectors/example-integrations/heroku)
* [Google Cloud Function](/data-connectors/example-integrations/google-cloud-functions)
* [AWS Lambda](/data-connectors/example-integrations/aws-lambda)
* [Azure HTTP Trigger](/data-connectors/example-integrations/azure-http-triggers)
* [IBM Cloud Actions](/data-connectors/example-integrations/ibm-cloud-actions)

## Supported Runtimes

If the provided example code does not contain your favorite runtime, use the following guides combined with the existing examples above, which should be enough to recreate the implementation yourself

* [Signing Events](/data-connectors/advanced-configurations#signing-events)
* [Verifying Signed Events](/data-connectors/receiving-events#verifying-signed-requests)


# Heroku

An example integration on forwarding Data Connector events to a server hosted on Heroku.

## Overview

This example uses a Data Connector to forward the events of all devices in a project to a server hosted on [Heroku](https://dashboard.heroku.com/apps). When receiving the HTTPS POST request, our application will verify both the origin and content of the request using a [Signature Secret](/data-connectors/advanced-configurations#signing-events).

## Prerequisites

The following points are assumed.

* You have the [role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher in your DT Studio project.
* You are familiar with the [Introduction to Data Connectors](/data-connectors/introduction-to-data-connector) and know how to [Create a Data Connector](/data-connectors/creating-a-data-connector).
* You are familiar with the [Heroku Developer Documentation](https://devcenter.heroku.com/).

## Heroku

Ensure you have the following software installed on your local machine.

* [Git](https://git-scm.com/), the popular version control tool.
* [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli), for interacting with Heroku.
* A functioning local environment for the language of your choice.

### Create a New App Locally

On your machine, create and enter a new directory for your app.

```
mkdir my-app
cd my-app
```

{% tabs %}
{% tab title="Python 3.11" %}
In the directory, create a new file `app.py` where you paste the following snippet which contains the Flask server code for receiving and validating the Data Connector request.

```python
import os
import hashlib

import jwt
from flask import Flask, request

app = Flask(__name__)

# Fetch environment variables.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')


def verify_request(body, token):
    # Decode the token using signature secret.
    try:
        payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
    except Exception as err:
        print(err)
        return False

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(body)
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        print('Checksum Mismatch')
        return False

    return True


@app.route('/', methods=['POST'])
def endpoint():
    # Extract necessary request information.
    body = request.data
    token = request.headers['x-dt-signature']

    # Validate request origin and content integrity.
    if not verify_request(body, token):
        return ('Could not verify request.', 400)

    # Print the request body.
    print(body)

    return ('OK', 200)
```

In the same directory, create a new file `requirements.txt` with the following contents. Heroku will install these dependencies in its environment before starting the server.

{% code title="requirements.txt" %}

```
gunicorn==20.1.0
flask==2.3.2
pyjwt==2.7.0
```

{% endcode %}

Finally, specify the command Heroku should run by adding a `Procfile` with the following snippet. This instructs Heroku that the server should run a single process of type `web`, where [Gunicorn](https://docs.gunicorn.org/en/stable/index.html) should be used to serve the app using a variable called `app` in a module called `app` (`app.py`)

{% code title="Procfile" %}

```
web: gunicorn app:app
```

{% endcode %}
{% endtab %}

{% tab title="Python API" %}
In the directory, create a new file `app.py` where you paste the following snippet which contains the Flask server code for receiving, validating, and decoding the Data Connector request.

{% code title="app.py" %}

```python
import os

from dtintegrations import data_connector, provider
from flask import Flask, request

app = Flask(__name__)


@app.route('/', methods=['POST'])
def endpoint():
    # Use the provider-specific validation function.
    payload = data_connector.HttpPush.from_provider(
        request=request,
        provider=provider.FLASK,
        secret=os.getenv('DT_SIGNATURE_SECRET'),
    )

    # Print the payload data.
    print(payload)

    # If all is well, return 200 response.
    return 'Success'
```

{% endcode %}

In the same directory, create a new file `requirements.txt` with the following contents. Heroku will install these dependencies in its environment before starting the server.

{% code title="requirements.txt" %}

```
gunicorn==20.1.0
flask==2.3.2
dtintegrations==0.5.1
```

{% endcode %}

Finally, specify the command Heroku should run by adding a `Procfile` with the following snippet. This instructs Heroku that the server should run a single process of type `web`, where [Gunicorn](https://docs.gunicorn.org/en/stable/index.html) should be used to serve the app using a variable called `app` in a module called `app` (`app.py`)

{% code title="Procfile" %}

```
web: gunicorn app:app
```

{% endcode %}
{% endtab %}

{% tab title="Node.js 16" %}
In the directory, initialize a node application with the following command. This generates the file `package.json` which contains information used by Heroku when deploying the app.

```
npm init
```

Create a new file `index.js` where you paste the following snippet which contains the Express server code for receiving and validating the Data Connector request.

```javascript
const express = require("express")
const bodyParser = require("body-parser")
const crypto = require('crypto')
const jwt = require('jsonwebtoken')

// Create the express server.
const app = express()
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())

// Fetch environment variables.
const signatureSecret = process.env.DT_SIGNATURE_SECRET

function verifyRequest(body, token) {
    // Decode the token using signature secret.
    let decoded;
    try {
        decoded = jwt.verify(token, signatureSecret)
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(JSON.stringify(body)).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

app.post("/", (req, res) => {
    // Extract necessary request information.
    let body  = req.body
    let token = req.get('X-Dt-signature')
    
    // Validate request origin and content integrity.
    if (verifyRequest(body, token) === false) {
        res.sendStatus(400)
        return
    } 

    // Log the request body.
    console.log(body)
    
    res.sendStatus(200)
})

app.listen(process.env.PORT)
```

In the same directory, install the following packages. This updates `package.json` so that Heroku can replicate the environment when deploying.

```
npm install body-parser==1.20.2
npm install express==4.18.2
npm install jsonwebtoken==9.0.1
```

Create a file `Procfile` with the following snippet. This instructs Heroku that the server should run a single process of type `web`, where `node index.js` should be used to serve the app.

```
web: node index.js
```

{% endtab %}

{% tab title="Go 1.20" %}
In the directory, initialize a go application with the following command. This generates the file `go.mod` which specifies version and requirements. Give any name you like.

```
go mod init example
```

Create a new file `main.go` where you paste the following snippet which contains the server code for receiving and validating the Data Connector request.

```go
package main

import (
	"crypto/sha1"
	"encoding/hex"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"

	jwt "github.com/golang-jwt/jwt/v5"
)

// Environment variables.
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")

// verifyRequest validates the request origin and content integrity.
func verifyRequest(bodyBytes []byte, tokenString string) error {
	// Decode the token using signature secret.
	claims := jwt.MapClaims{}
	_, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
		return []byte(signatureSecret), nil
	})
	if err != nil {
		return err
	}

	// Verify the request body checksum.
	sha1Bytes := sha1.Sum(bodyBytes)
	sha1String := hex.EncodeToString(sha1Bytes[:])
	if sha1String != claims["checksum"] {
		return fmt.Errorf("Checksum mismatch.")
	}

	return nil
}

func endpoint(w http.ResponseWriter, r *http.Request) {
	if r.URL.Path != "/" {
		http.Error(w, "404 not found.", http.StatusNotFound)
		return
	}

	// Take an action depending on the request method.
	switch r.Method {
	case "POST":
		if err := r.ParseForm(); err != nil {
			fmt.Fprintf(w, "ParseForm() err: %v", err)
			return
		}

		// Attempt to extract the body bytes of the request.
		body, err := ioutil.ReadAll(r.Body)
		if err != nil {
			log.Printf("Error reading body: %v", err)
			http.Error(w, "Can't read body", http.StatusBadRequest)
			return
		}

		// Validate the origin and contents of the request.
		if err = verifyRequest(body, r.Header.Get("X-Dt-Signature")); err != nil {
			log.Printf("Could not validate request: %v", err)
			http.Error(w, "Could not validate request", http.StatusBadRequest)
			return
		}

		//
		// Further processing here.
		//

		log.Println("Success")

	default:
		http.Error(w, "Only POST methods are allowed.", http.StatusMethodNotAllowed)
	}
}

func main() {
	http.HandleFunc("/", endpoint)

	port := ":" + string(os.Getenv("PORT"))
	log.Printf("Listening on port %s.", port)
	if err := http.ListenAndServe(port, nil); err != nil {
		log.Fatal(err)
	}
}
```

In the same directory, run the following command to install the required package.

```
go get github.com/golang-jwt/jwt/v5@v5.0.0
```

Build your application into a new directory `bin`.

```
GOOS=linux GOARCH=amd64 go build -o bin/main -v .
```

Finally, specify the command Heroku should run by adding a `Procfile` with the following snippet. This instructs Heroku that the server should run a single process of type `web`, where the app should be server from the binary `bin/main` that we built in the previous step.

{% code title="Procfile" %}

```
web: bin/main
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Deploy the App

The following steps will deploy your application to a new app in Heroku. Take note of the **app name** given to the application in step 4.

1. Initialize your app's directory as a Git repository: `git init`
2. Add all changes: `git add .`
3. Commit the changes: `git commit -m "initial commit"`
4. Create the Heroku app, and add it as a Git remote: `heroku create <APP_NAME>`. You can pick a unique app name yourself or let Heroku create a random one by omitting `<APP_NAME>`. This will be used as a part of the URL for your server.&#x20;
5. Push your application to Heroku: `git push heroku main`

### Post Deployment Configuration

Add a new signature secret configuration variable to your app using the following command. This can be read by the python instance and imported as an environment variable to use. Use a strong secret.

```
heroku config:set -a <APP_NAME> DT_SIGNATURE_SECRET=<YOUR_SECRET>
```

Your app is now ready to receive requests, but we need to know the endpoint URL. This can be acquired by looking at the **Web URL** field in the results of the following command. Save it for later.

```
heroku apps:info -a <APP_NAME>
```

## Create a Data Connector

To continuously forward the data to our newly created Heroku server, a Data Connector with almost all default settings is sufficient. Refer to [Creating a Data Connector](/data-connectors/creating-a-data-connector) if you are unfamiliar with the process. The following configurations should be set.

* **Endpoint URL:** The **Web URL** found in the previous step.
* **Signature Secret:** The value of **DT\_SIGNATURE\_SECRET** environment variable.

![](/files/-MbaxD1X7ovqWtZ_UjOp)

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the [NetworkStatusEvent](/concepts/events#network-status-event) is sent every [Periodic Heartbeat](/concepts/events#periodic-heartbeat) and will by default be forwarded by the Data Connector if not explicitly unticked.

## Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each [Periodic Heartbeat](/concepts/events#periodic-heartbeat) or by touching a sensor to force a new event.

![](/files/-MRtoUIHpVcTjEl8cyYJ)

If instead the Error counter increments, a response containing a non-2xx status code is returned.

* Verify that the Data Connector endpoint URL is correct.
* You can view the logs of your Heroku app with the following command.

  ```
  heroku logs --app=<APP_NAME> --tail
  ```


# Google Cloud Functions

An example integration where a Data Connector forwards events to a Google Cloud Function.

## Overview

This example uses a Data Connector to forward the events of all devices in a project to a [Google Cloud Function](https://cloud.google.com/functions). When receiving the HTTPS POST request, our function will verify both the origin and content of the request using a [signature secret](/data-connectors/advanced-configurations#signing-events), then decode the data.

## Prerequisites

The following points are assumed.

* You have the [role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher in your DT Studio project.
* You are familiar with the [Introduction to Data Connectors](/data-connectors/introduction-to-data-connector) and know how to [Create a Data Connector](/data-connectors/creating-a-data-connector).
* You are familiar with the [Cloud Functions documentation](https://cloud.google.com/functions/docs).

## Google Cloud Platform

While there are many advantages to using a local environment for development, this guide will be using the browser portal to minimize setup requirements.

### Create a Cloud Function

[Following this guide](https://cloud.google.com/functions/docs/console-quickstart), create a new Cloud Function with the following configurations.

{% tabs %}
{% tab title="Python 3.11" %}

### (1) Configuration

#### Basics

* **Environment:** 2nd gen
* **Function name:** As desired.
* **Region:** As desired.

#### Trigger HTTPS

* **Authentication:** Allow unauthenticated invocations.

**Runtime, build, connections, and security settings**

Add a new runtime environment variable with the following values.

* **Name:** `DT_SIGNATURE_SECRET`
* **Value:** A unique password. We will need it later, so write it down.

### (2) Code

* **Runtime:** Python 3.11
* **Entry point:** `dataconnector_endpoint`

In the Source Code, edit **main.py** with the following snippet. The implementation is explained in detail on the [Data Connector Advanced Configurations](/data-connectors/advanced-configurations#signing-events) page.

{% code title="main.py" %}

```python
import os
import hashlib

import jwt
import functions_framework

# Fetch environment variable secret.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')


def verify_request(body, token):
    """
    Verifies that the request originated from DT and that the body
    hasn't been modified since it was sent. This is done by verifying
    that the checksum field of the JWT token matches the checksum of the
    request body and that the JWT is signed with the signature secret.
    """

    # Decode the token using signature secret.
    try:
        payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
    except Exception as err:
        print(err)
        return False

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(body)
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        print('Checksum Mismatch')
        return False

    return True


@functions_framework.http
def dataconnector_endpoint(request):
    # Extract the body as a byte string and the signed JWT.
    # We'll use these values to verify the request.
    payload = request.get_data()
    token = request.headers['x-dt-signature']

    # Verify request origin and content integrity.
    if not verify_request(payload, token):
        return ('Could not verify request.', 400)

    # Decode the body as JSON
    body = request.get_json(silent=True)

    # Fetch some information about the event, then print.
    # You can implement your own logic here as desired.
    event_type: str = body['event']['eventType']
    device_type: str = body['metadata']['deviceType']
    device_id: str = body['metadata']['deviceId']
    print(f'Got {event_type} event from {device_type} device {device_id}.')

    return ('OK', 200)
```

{% endcode %}

Edit **requirements.txt** with the following entries.

{% code title="requirements.txt" %}

```
functions-framework==3.*
pyjwt==2.7.0
```

{% endcode %}
{% endtab %}

{% tab title="Python API" %}

### (1) Configuration

#### Basics

* **Environment:** 2nd gen
* **Function name:** As desired.
* **Region:** As desired.

#### Trigger HTTPS

* **Authentication:** Allow unauthenticated invocations.

**Runtime, build, connections, and security settings**

Add a new runtime environment variable with the following values.

* **Name:** `DT_SIGNATURE_SECRET`
* **Value:** A unique password. We will need it later, so write it down.

### (2) Code

* **Runtime:** Python 3.11
* **Entry point:** `dataconnector_endpoint`

In the Source Code, edit **main.py** with the following snippet.\
For details, read our [Python API Documentation](https://developer.disruptive-technologies.com/api/libraries/python/).

{% code title="main.py" %}

```python
import os
import functions_framework

from dtintegrations import data_connector, provider


@functions_framework.http
def dataconnector_endpoint(request):
    # Validate and decode the incoming request.
    payload = data_connector.HttpPush.from_provider(
        request=request,
        provider=provider.GCLOUD,
        secret=os.getenv('DT_SIGNATURE_SECRET'),
    )

    # Print the payload data.
    print(payload)

    # If all is well, return a 200 response.
    return ('OK', 200)
```

{% endcode %}

Edit **requirements.txt** with the following entries.

{% code title="requirements.txt" %}

```
functions-framework==3.*
dtintegrations==0.5.1
```

{% endcode %}
{% endtab %}

{% tab title="Node.js 20" %}

### (1) Configuration

#### Basics

* **Environment:** 2nd gen
* **Function name:** As desired.
* **Region:** As desired.

#### Trigger HTTPS

* **Authentication:** Allow unauthenticated invocations.

**Runtime, build, connections, and security settings**

Add a new runtime environment variable with the following values.

* **Name:** `DT_SIGNATURE_SECRET`
* **Value:** A unique password. We will need it later, so write it down.

### (2) Code

* **Runtime:** Node.js 20
* **Entry point:** `dataconnectorEndpoint`

In the Source Code, edit **index.js** with the following snippet. The implementation is explained in detail on the [Data Connector Advanced Configurations](/data-connectors/advanced-configurations#signing-events) page.

{% code title="index.js" %}

```javascript
const crypto = require('crypto')
const jwt = require('jsonwebtoken')
const functions = require('@google-cloud/functions-framework')

// Fetch environment variables
const signatureSecret = process.env.DT_SIGNATURE_SECRET

// Verifies that the request originated from DT, and that the body
// hasn't been modified since it was sent. This is done by verifying
// that the checksum field of the JWT token matches the checksum of the
// request body, and that the JWT is signed with the signature secret.
function verifyRequest(body, token) {
    // Decode the JWT, and verify that it was 
    // signed using the signature secret.
    let decoded
    try {
        decoded = jwt.verify(token, signatureSecret)
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(body).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

functions.http('dataconnectorEndpoint', (req, res) => {
    // Extract the body as a string and the signed JWT.
    // We'll use these values to verify the request. 
    let payload  = JSON.stringify(req.body)
    let token = req.get('X-Dt-Signature')
    
    // Validate request origin and content integrity.
    if (verifyRequest(payload, token) === false) {
        res.sendStatus(400)
        return
    }
    
    // First, check if the event type is one of the event
    // types we're expecting. 
    // As an example, we'll check for touch events here.
    switch (req.body.event.eventType) {
        case 'touch':
            // Now that we know this is a device event, we can 
            // check for the device type and device identifier 
            // in the event metadata.
            deviceType = req.body.metadata.deviceType
            deviceId = req.body.metadata.deviceId

            console.log(`Received touch event from ${deviceType} sensor with id ${deviceId}`)
            break
        default:
            break
    }

    res.sendStatus(200);
});
```

{% endcode %}

Edit **package.json** to contain the following dependencies field.

{% code title="package.json" %}

```javascript
{
  "dependencies": {
    "@google-cloud/functions-framework": "^3.0.0",
    "jsonwebtoken": "^9.0.1"
  }
}
```

{% endcode %}
{% endtab %}

{% tab title="Go 1.20" %}

### (1) Configuration

#### Basics

* **Environment:** 2nd gen
* **Function name:** As desired.
* **Region:** As desired.

#### Trigger HTTPS

* **Authentication:** Allow unauthenticated invocations.

**Runtime, build, connections, and security settings**

Add a new runtime environment variable with the following values.

* **Name:** `DT_SIGNATURE_SECRET`
* **Value:** A unique password. We will need it later, so write it down.

### (2) Code

* **Runtime:** Go 1.20
* **Entry point:** `DataconnectorEndpoint`

In the Source Code, edit **function.go** with the following snippet. The implementation is explained in detail on the [Data Connector Advanced Configurations](/data-connectors/advanced-configurations#signing-events) page.

{% code title="function.go" %}

```go
package dconendpoint

import (
    "crypto/sha1"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"

    "github.com/GoogleCloudPlatform/functions-framework-go/functions"
    jwt "github.com/golang-jwt/jwt/v5"
)

func init() {
    functions.HTTP("DataconnectorEndpoint", DataconnectorEndpoint)
}

// Environment variables.
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")

// verifyRequest verifies that the request originated from DT and that the
// body hasn't been modified since it was sent. This is done by verifying
// that the checksum field of the JWT token matches the checksum of the
// request body, and that the JWT is signed with the signature secret.
func verifyRequest(bodyBytes []byte, tokenString string) error {
    // Decode the JWT, and verify that it was signed using the signature secret.
    // Also verifies the algorithm used to sign the JWT.
    token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
        // Return out signature secret to verify that it was used to sign the JWT.
        return []byte(signatureSecret), nil
    }, jwt.WithValidMethods([]string{"HS256"}))
    if err != nil {
        return err
    }

    // Verify the request body checksum.
    sha1Bytes := sha1.Sum(bodyBytes)
    sha1String := hex.EncodeToString(sha1Bytes[:])
    claims := token.Claims.(jwt.MapClaims)
    if sha1String != claims["checksum"] {
        return fmt.Errorf("Checksum mismatch.")
    }

    return nil
}

// DataConnectorEndpoint receives, validates, and returns a response for the forwarded event.
func DataconnectorEndpoint(w http.ResponseWriter, r *http.Request) {
    // Extract the body and the signed JWT.
    // We'll use these values to verify the request.
    tokenString := r.Header.Get("x-dt-signature")
    bodyBytes, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Println(err)
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // Validate request origin and content integrity.
    if err := verifyRequest(bodyBytes, tokenString); err != nil {
        fmt.Println(err)
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // We now know the request came from DT Cloud and the integrity
    // of the body has been verified. We can now handle the event safely.
    if err := handleEvent(bodyBytes); err != nil {
        fmt.Println(err)
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // Respond with a 200 status code to ack the event,
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("OK"))
}

func handleEvent(payload []byte) error {
    // The structure of the events we'll receive from a Data Connector.
    type Event struct {
        Event struct {
            EventId   string          `json:"eventId"`
            EventType string          `json:"eventType"`
            Data      json.RawMessage `json:"data"`
            Timestamp string          `json:"timestamp"`
        } `json:"event"`
        Labels   map[string]string `json:"labels"`
        Metadata map[string]string `json:"metadata"`
    }

    // The structure of the `Event.Data` field for a touch event.
    // We'll be using touch events for this example.
    type TouchData struct {
        Touch struct {
            Timestamp string `json:"updateTime"`
        } `json:"touch"`
    }

    // Decode the event
    var event Event
    if err := json.Unmarshal(payload, &event); err != nil {
        return err
    }

    // First, check if the event type is one of the
    // event-types we're expecting.
    // As an example, we'll check for touch events here.
    switch event.Event.EventType {
    case "touch":
        // Now that we know this is a touch event, we can decode
        // the `Event.Data` field.
        var touchData TouchData
        if err := json.Unmarshal(event.Event.Data, &touchData); err != nil {
            return err
        }

        // Also, since we now know this is a device event, we can
        // check for the device type and device identifier
        // in the event metadata.
        deviceType := event.Metadata["deviceType"]
        deviceId := event.Metadata["deviceId"]
        timestamp := touchData.Touch.Timestamp

        fmt.Printf("Received touch event at %s from %s sensor with id %s\n",
            timestamp,
            deviceType,
            deviceId,
        )
    }

    return nil
}

```

{% endcode %}

Replace the content of **go.mod** with the following snippet.

```
module example.com/gcf

require (
	github.com/GoogleCloudPlatform/functions-framework-go v1.5.2
	github.com/golang-jwt/jwt/v5 v5.0.0
)
```

{% endtab %}
{% endtabs %}

When configured, **deploy** your function.

### Post Deployment

Your function is now ready to receive requests, but we need to know the target URL. In your function, locate the **TRIGGER** tab and copy the **Trigger URL**. Save this for later.

## Create a Data Connector

To continuously forward the data to our newly created Cloud Function, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our [Creating a Data Connector](/data-connectors/creating-a-data-connector) guide using the following configurations.

* **Endpoint URL:** The **Trigger URL** we found in the previous step.
* **Signature Secret:** The value of the **DT\_SIGNATURE\_SECRET** environment variable.

![](/files/-MRto5TBuJfY3eA2vB6_)

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the `networkStatus` event is sent every [periodic heartbeat](/concepts/events#periodic-heartbeat) and will by default be forwarded by the Data Connector if not explicitly unticked.

## Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each [periodic heartbeat](/concepts/events#periodic-heartbeat) or by touching a sensor to force a new event.

![](/files/-MRtoUIHpVcTjEl8cyYJ)

If instead the Error counter increments, a response containing a non-2xx status code is returned.

* Verify that the Data Connector endpoint URL is correct.
* Google provides a host of tools that [monitor Cloud Functions](https://cloud.google.com/functions/docs/monitoring).  Check the logs for any tracebacks that could explain why an error is returned.

## Next steps

Your sensor data is now in the Google Cloud environment, and you can start using it in their various services. Fortunately, Google has some well-documented guides to get you started.

### PostgreSQL Database

A database should be tailored to each specific use case. However, if you're uncertain, PostgreSQL (Postgres) is a good place to get started. The following guides will show you how to create a new Postgres database, then connect your Cloud Function to execute queries.

* [Quickstart for Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres/quickstart).
* [Connecting from Cloud Functions to Cloud SQL](https://cloud.google.com/sql/docs/mysql/connect-functions#python).

### DataStudio

Once your Database has been set up, the following guide shows you how to connect it to [DataStudio](https://datastudio.google.com/) for continuous data visualization and analytics.

* [Connect Data Studio to Cloud SQL](https://support.google.com/datastudio/answer/7020436?hl=en).


# AWS Lambda

An example integration where a Data Connector forwards events to an AWS Lambda Function.

## Overview

This example uses a Data Connector to forward the events of all devices in a project to an [AWS Lambda Function](https://aws.amazon.com/lambda/). When receiving the HTTPS POST request, our function will verify both the origin and content of the request using a [Signature Secret](/data-connectors/advanced-configurations#signing-events), then decode the data.

## Prerequisites

We will create, configure, and deploy our Lambda function using the AWS Console, but the deployment package that contains the code and related runtime must be built locally on your machine. You must therefore have a working development environment.

The following points are assumed.

* You have a [Role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher in your DT Studio project.
* You are familiar with the [Introduction to Data Connectors](/data-connectors/introduction-to-data-connector) and know how to [Create a Data Connector](/data-connectors/creating-a-data-connector).
* You are familiar with the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html).

## Create a Local Deployment Package

AWS Lambda Functions can be created and deployed using either a Docker Image or a .zip archive. Both will achieve the same result, but we will here use a .zip archive in the interest of simplicity.

### Source Code

Create and enter a new root directory in which your function will be developed.

```bash
mkdir my-lambda-function
cd my-lambda-function
```

{% tabs %}
{% tab title="Python 3.10" %}
In your directory root, create a new file **lambda\_function.py** with the following content. The implementation is explained in detail on the [Data Connector Advanced Configurations](/data-connectors/advanced-configurations#signing-events) page.

```python
import os
import json
import hashlib
import jwt

# Fetch secret environment variable.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')


def verify_request(body, token):
    # Decode the token using signature secret.
    payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(body.encode('ascii'))
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        raise ValueError('Checksum Mismatch')


def lambda_handler(event, context):
    # Extract necessary request information.
    body = event['body']
    token = event['headers']['X-Dt-Signature']

    # Validate request origin and content integrity.
    try:
        verify_request(body, token)
    except Exception as e:
        return {
            'statusCode': 400,
            'body': json.dumps(str(e)),
        }

    #
    # Further processing here.
    #

    return {
        'statusCode': 200,
        'body': json.dumps('Success'),
    }
```

{% endtab %}

{% tab title="Node.js 18" %}
In your directory root, create a file **index.js** in which you paste the following snippet. The implementation is explained in detail on the [Data Connector Advanced Configurations](/data-connectors/advanced-configurations#signing-events) page.

```javascript
var crypto = require('crypto');
var jwt = require('jsonwebtoken');

// Fetch environment variables.
const signatureSecret = process.env.DT_SIGNATURE_SECRET

function verifyRequest(body, token) {
    // Decode the token using signature secret.
    let decoded;
    try {
        decoded = jwt.verify(token, signatureSecret)
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(JSON.stringify(body)).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

exports.handler = async function(event, context) {
    // Extract necessary request information.
    let body  = JSON.parse(event.body)
    let token = event.multiValueHeaders['X-Dt-Signature'][0]

    // Validate request origin and content integrity.
    if (verifyRequest(body, token) === false) {
        const response = {
            statusCode: 400,
            body: JSON.stringify('could not verify request'),
        };
        return response; 
    }

    //
    // Further processing here.
    //
    const response = {
        statusCode: 200,
        body: JSON.stringify('success'),
    };
    return response; 
};
```

{% endtab %}

{% tab title="Go 1.x" %}
In your directory root, initialize a new go module.

```
go mod init example
```

Create a file **main.go** in which you paste the following snippet.

```go
package main

import (
	"context"
	"crypto/sha1"
	"encoding/hex"
	"log"
	"os"

	jwt "github.com/dgrijalva/jwt-go"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
)

// Environment variables.
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")

// verifyRequest validates the request origin and content integrity.
func verifyRequest(bodyBytes []byte, tokenString string) bool {
	// Decode the token using signature secret.
	claims := jwt.MapClaims{}
	_, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
		return []byte(signatureSecret), nil
	})
	if err != nil {
		log.Println(err)
		return false
	}

	// Verify the request body checksum.
	sha1Bytes := sha1.Sum(bodyBytes)
	sha1String := hex.EncodeToString(sha1Bytes[:])
	if sha1String != claims["checksum"] {
		log.Println("Checksum mismatch.")
		return false
	}

	return true
}

func handleRequest(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	// Extract necessary request information.
	tokenString := req.Headers["X-Dt-Signature"]
	bodyBytes := []byte(req.Body)

	// Validate request origin and content integrity.
	if !verifyRequest(bodyBytes, tokenString) {
		return events.APIGatewayProxyResponse{StatusCode: 400}, nil
	}

	return events.APIGatewayProxyResponse{StatusCode: 200}, nil
}

func main() {
	lambda.Start(handleRequest)
}
```

{% endtab %}
{% endtabs %}

### Dependencies and Compilation

From inside your function root directory, install the following dependencies.

{% tabs %}
{% tab title="Python 3.10" %}

```
pip install --target . pyjwt==2.7.0
```

Your directory tree should now look something like this.

```
.
├── lambda_function.py/
├── jwt/
└── PyJWT-2.7.0.dist-info
```

{% endtab %}

{% tab title="Node.js 18" %}

```bash
npm install aws-xray-sdk@3.5.0
npm install jsonwebtoken@9.0.1
```

Your directory tree should now look something like this.

```bash
.
├── index.js
├── node_modules
│   ├── aws-xray-sdk
│   ├── jsonwebtoken
│   ├── ...
│   ├── ...
├── package.json
└── package-lock.json
```

{% endtab %}

{% tab title="Go 1.x" %}

```
go get github.com/aws/aws-lambda-go@v1.41.0
go getgithub.com/dgrijalva/jwt-go@v3.2.0
```

Your directory tree should now look something like this.

```
.
├── go.mod
├── go.sum
└── main.go
```

{% endtab %}
{% endtabs %}

### Create a .zip Archive

The following steps create the deployment package .zip archive.

{% tabs %}
{% tab title="Python 3.10" %}
From your directory root, create the deployment package with the following command.

```bash
zip -r deployment-package.zip .
```

{% endtab %}

{% tab title="Node.js 18" %}
From your directory root, create the deployment package with the following command.

```bash
zip -r deployment-package.zip .
```

{% endtab %}

{% tab title="Go 1.x" %}
The following steps are for Linux and MacOS. If you're on Windows or want more details about the process, refer to the [official AWS Guide](https://docs.aws.amazon.com/lambda/latest/dg/golang-package.html).

Compile your executable.

```
GOOS=linux GOARCH=amd64 go build main.go
```

From your directory root, create the deployment package with the following command.

```
zip deployment-package.zip main
```

{% endtab %}
{% endtabs %}

## Create a New Lambda Function

We are now ready to create our new Lambda function. In the AWS web Console, find the Lambda service and click **Create function** and set the following parameters.

{% tabs %}
{% tab title="Python 3.10" %}
Select **Author from scratch** and set the following **Basic Information**.

* **Function name:** As desired.
* **Runtime:** Python 3.10
  {% endtab %}

{% tab title="Node.js 18" %}
Select **Author from scratch** and set the following **Basic Information**.

* **Function name:** As desired.
* **Runtime:** Node.js 18
  {% endtab %}

{% tab title="Go 1.x" %}
Select **Author from scratch** and set the following **Basic Information**.

* **Function name:** As desired.
* **Runtime:** Go 1.x
  {% endtab %}
  {% endtabs %}

Click **Create function** to move onward to configuration.

## Configure Lambda Function

Anything not mentioned here can be left default or changed as desired.

### New Trigger

Under the **Configuration** tab, click **Triggers** and then **Add trigger**. Choose the **API Gateway** source with the following parameters.

* **Intent:** Create a new API
* **API type:** HTTP API
* **Security:** Open

Click **Save** and note the URL. We will need it later.

### Environment Variables

Under the **Configuration** tab, click **Environment variables** and then **Edit**. Add a signature secret.

* **Name:** `DT_SIGNATURE_SECRET`
* **Value:** A unique password. We will need it later, so write it down.

Leaving **Encryption configuration** default will let AWS handle encryption at rest, but you can choose to handle the master key if you so desire.

## Create a New Data Connector

To continuously forward the data to our newly created Lambda Function, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our [Creating a Data Connector](/data-connectors/creating-a-data-connector) guide. The following configurations should be set.

* **Endpoint URL:** The **API endpoint URL** found in the previous step.
* **Signature Secret:** The value of **DT\_SIGNATURE\_SECRET** parameter set earlier.

![](/files/-MRto5TBuJfY3eA2vB6_)

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.

## Test the Integration <a href="#test-the-integration" id="test-the-integration"></a>

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each [Periodic Heartbeat](https://app.gitbook.com/@disruptive/s/developer-docs/~/drafts/-MSSn8rMbsoOFUC9VGeL/references/events#periodic-heartbeat) or by touching a sensor to force a new event.

![](/files/-MSDNrkr4EHaUlFAWn6A)

If instead the Error counter increments, a response containing a non-200 status code is returned.

* Verify that the Data Connector endpoint URL is correct.
* AWS provides a host of tools that can be used to [monitor Lambda Functions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-monitoring.html).  Check the logs for any tracebacks that could explain why an error is returned.

## Next steps

Your sensor data is now in the AWS environment, and you can start using it in their various services. Fortunately, AWS has some well-documented guides to get you started.

### PostgreSQL Database

A database should be tailored to each specific use case. However, if you're uncertain, PostgreSQL (Postgres) is a good place to get started. The following guides will show you how to create a new Postgres database, then connect your Lambda Function to execute queries.

* [Create and Connect to a PostgreSQL Database](https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/).
* [Configuring a Lambda Function to Access Amazon RDS](https://docs.aws.amazon.com/lambda/latest/dg/services-rds-tutorial.html).


# Azure HTTP Triggers

An example integration where a Data Connector forwards events to an Azure HTTP Trigger.

## Overview

This example uses a Data Connector to forward the events of all devices in a project to an [Azure HTTP trigger](https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-http-webhook-trigger?tabs=csharp). When receiving the HTTPS POST request, our trigger will verify both the origin and content of the request using a [Signature Secret](/data-connectors/advanced-configurations#signing-events), then decode the data.

## Prerequisites

The following points are assumed.

* You have a [Role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher in your DT Studio project.
* You are familiar with the [Introduction to Data Connectors](/data-connectors/introduction-to-data-connector) and know how to [Create a Data Connector](/data-connectors/creating-a-data-connector).
* You are familiar with the [Azure Functions documentation](https://docs.microsoft.com/en-us/azure/azure-functions/).

## Azure Function

Depending on the runtime, parts of creating the HTTP Trigger must be done in a local environment on your machine. Ensure you have a working runtime environment for development using either the [command line](https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-node?tabs=azure-cli%2Cbrowser) or [Visual Studio Code](https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-vs-code-node).

### Create a Function App

Function Apps combine several functions into a single unit for easier management and can be created in the Azure Portal by [following this guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-function-app-portal#create-a-function-app). Create a new Function App with the following configurations.

{% tabs %}
{% tab title="Python 3.11" %}

* **Runtime Stack:** Python
* **Version:** 3.11
* **Operating System:** Linux
  {% endtab %}

{% tab title="Python API" %}

* **Runtime Stack:** Python
* **Version:** 3.11
* **Operating System:** Linux
  {% endtab %}

{% tab title="Node.js 18" %}

* **Runtime Stack:** Node.js
* **Version:** 18 LTS
* **Operating System:** Linux
  {% endtab %}
  {% endtabs %}

Click **Review + create**, then **Create** and wait for deployment. Once deployed, **Go to resource**.

#### Edit Application Settings

We will verify the incoming POST request by a signature secret, as explained in the Data Connector [Advanced Configurations](/data-connectors/advanced-configurations#signing-events). In your new Function App, add a secret to your App environment by adding a new Application Setting, [as explained in this guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-how-to-use-azure-function-app-settings?tabs=portal#settings).

* **Name:** DT\_SIGNATURE\_SECRET
* **Value:** A unique password. We will use it later, so write it down.

### Create an HTTP Trigger Function

Once the Function App is deployed, a new HTTP trigger function can be created. Azure has guides on creating and editing functions. Follow either.

* [command line](https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?tabs=azure-cli%2Cbash%2Cbrowser)
* [Visual Studio Code](https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-vs-code-python)

By following one of the guides, create a new HTTP Trigger with the following configurations.

* **Authorization level:** Anonymous.

Afterward, your project structure should end up looking something like this.

{% tabs %}
{% tab title="Python 3.11" %}

```bash
project
├── yourNewFunction
│   ├── function.json
│   └── __init__.py
├── getting_started.md
├── host.json
├── local.settings.json
└── requirements.txt
```

{% endtab %}

{% tab title="Python API" %}

```bash
project
├── yourNewFunction
│   ├── function.json
│   └── __init__.py
├── getting_started.md
├── host.json
├── local.settings.json
└── requirements.txt
```

{% endtab %}

{% tab title="Node.js 18" %}

```bash
project
├── host.json
├── yourNewFunction
│   ├── function.json
│   └── index.js
├── local.settings.json
└── package.json
```

{% endtab %}
{% endtabs %}

### Modify the HTTP Trigger

Once the function has been created, the desired behavior can be programmed. The following code snippet implements a simple HTTP trigger that listens for POST requests, then validates the content and origin using the Signature Secret and Checksum verification.

{% tabs %}
{% tab title="Python 3.11" %}
In your HTTP Trigger function directory, edit the file `__init__.py`. The implementation is explained in detail on the [Data Connector Receiving Events](/data-connectors/receiving-events#verifying-signed-events) page.

{% code title="**init**.py" %}

```python
import os
import hashlib

import azure.functions as func
import jwt

# Fetch environment variables.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET', '')


def verify_request(body, token):
    # Decode the token using signature secret.
    try:
        payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
    except Exception as err:
        print(err)
        return False

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(body)
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        print('Checksum Mismatch')
        return False

    return True


def main(req: func.HttpRequest) -> func.HttpResponse:
    # Extract necessary request information.
    body = req.get_body()
    token = req.headers['x-dt-signature']

    # Validate request origin and content integrity.
    if not verify_request(body, token):
        return func.HttpResponse('Bad request', status_code=400)

    #
    # Further processing goes here.
    #

    return func.HttpResponse("OK", status_code=200)
```

{% endcode %}
{% endtab %}

{% tab title="Python API" %}
In your HTTP Trigger function directory, edit the file `__init__.py`. The implementation is explained in detail on the [Data Connector Receiving Events](/data-connectors/receiving-events#verifying-signed-events) page.

{% code title="**init**.py" %}

```python
import os

import azure.functions as func
from dtintegrations import data_connector, provider


def main(req: func.HttpRequest) -> func.HttpResponse:
    # Extract necessary request information.
    payload = data_connector.HttpPush.from_provider(
        request=req,
        provider=provider.AZURE,
        secret=os.environ.get('DT_SIGNATURE_SECRET', ''),
    )

    #
    # Further processing goes here.
    #
    print(payload)

    return func.HttpResponse("OK", status_code=200)
```

{% endcode %}
{% endtab %}

{% tab title="Node.js 18" %}
In your HTTP Trigger function directory, edit the file `index.js`. The implementation is explained in detail on the [Data Connector Receiving Events](/data-connectors/receiving-events#verifying-signed-events) page.

{% code title="index.js" %}

```javascript
const crypto = require('crypto')
const jwt = require('jsonwebtoken')

// Fetch environment variables.
const signatureSecret = process.env.DT_SIGNATURE_SECRET

function verifyRequest(body, token) {
    // Decode the token using signature secret.
    let decoded;
    try {
        decoded = jwt.verify(token, signatureSecret)
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(JSON.stringify(body)).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

module.exports = async function (context, req) {
    // Extract necessary request information.
    let body  = req.body
    let token = req.headers['x-dt-signature']

    // Validate request origin and content integrity.
    if (verifyRequest(body, token) === false) {
        context.res = { status: 400, body: 'Bad request' }
        return
    }
    //
    // Further processing here.
    //
    
    context.res = { status: 200, body: 'OK' }
}
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Dependency Management

Dependencies must be specified and included for your HTTP Trigger to run in the cloud.

{% tabs %}
{% tab title="Python 3.11" %}
As explained [here](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python#package-management), edit `requirements.txt` to tell Azure which dependencies to install.

{% code title="requirements.txt" %}

```
azure-functions
pyjwt==2.7.0
```

{% endcode %}
{% endtab %}

{% tab title="Python API" %}
As explained [here](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python#package-management), edit `requirements.txt` to tell Azure which dependencies to install.

{% code title="requirements.txt" %}

```
azure-functions
dtintegrations==0.5.1
```

{% endcode %}
{% endtab %}

{% tab title="Node.js 18" %}
As explained [here](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-node?tabs=v2#deploying-with-dependencies), in your project directory, install all requisite packages locally using `npm install`. This will generate a directory **node\_modules** which is included when deploying.

```bash
npm install jsonwebtoken@9.0.1
```

{% endtab %}
{% endtabs %}

### Deploy

Your function should now be ready for deployment. It is considered good practice to test- and experiment with your function locally, [which you can read how-to here](https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-node?tabs=azure-cli%2Cbrowser#run-the-function-locally). The following command will deploy your function to the Function App created in the beginning.

```bash
func azure functionapp publish <APP_NAME>
```

Once the function is deployed, the **Invoke URL** is displayed in the output. Copy and save it as we will need it when creating our Data Connector.

```
...
Deployment successful.
Remote build succeeded!
Syncing triggers...
Functions in msdocs-azurefunctions-qs:
    HttpExample - [httpTrigger]
THIS->  Invoke url: https://msdocs-azurefunctions-qs.azurewebsites.net/api/httpexample
```

## Data Connector

To continuously forward the data to our newly created Cloud Function, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our [Creating a Data Connector](/data-connectors/creating-a-data-connector) guide. The following configurations should be set.

* **Endpoint URL:** The **Invoke URL** found in the previous step.
* **Signature Secret:** The value of **DT\_SIGNATURE\_SECRET** environment variable set earlier.

![](/files/-MRto5TBuJfY3eA2vB6_)

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.

## Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each [Periodic Heartbeat](/concepts/events#periodic-heartbeat) or by touching a sensor to force a new event.

![](/files/-MRtoUIHpVcTjEl8cyYJ)

If instead the Error counter increments, a response containing a non-200 status code is returned.

* Verify that the Data Connector endpoint URL is correct.
* Azure provides a host of tools that can be used to [monitor Azure Functions](https://docs.microsoft.com/en-us/azure/azure-functions/functions-monitoring#streaming-logs).  Check the logs for any tracebacks that could explain why an error is returned.

## Next steps

Your sensor data is now in the Azure environment, and you can start using it in their various services. Fortunately, Azure has some well-documented guides to get you started.

### PostgreSQL Database

A database should be tailored to each specific use case. However, if you're uncertain, PostgreSQL (Postgres) is a good place to get started. The following guides will show you how to create a new Postgres database, then connect your HTTP Trigger to execute queries.

* [Create an Azure Database for PostgreSQL server](https://docs.microsoft.com/en-us/azure/postgresql/quickstart-create-server-database-portal).
* [Connect and query data in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/connect-python).&#x20;


# IBM Cloud Actions

An example integration where a Data Connector forwards events to an IBM Cloud Action.

## Overview

This example uses a Data Connector to forward the events of all devices in a project to an [IBM Cloud Action](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-getting-started). When receiving the HTTPS POST request, our action will verify both the origin and content of the request using a [Signature Secret](/data-connectors/advanced-configurations#signing-events), then decode the data.

## Prerequisites

The following points are assumed.

* You have a [Role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher in your DT Studio project.
* You are familiar with the [Introduction to Data Connectors](/data-connectors/introduction-to-data-connector) and know how to [Create a Data Connector](/data-connectors/creating-a-data-connector).
* You are familiar with the [IBM Cloud Functions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-getting-started) documentation.

## IBM Cloud Platform

While there are many advantages to using a local environment for development, this guide will use the browser portal to minimize setup requirements.

### Create a Cloud Action

In the [cloud functions dashboard](https://cloud.ibm.com/functions/), create and deploy a new Action with one of the following runtimes.

* Python 3.11
* Node.js 16

### Edit Source Code

Once deployed, replace the code of your new Action with that matching your runtime. The implementation is explained in detail on the [Data Connector Receiving Events](/data-connectors/receiving-events#verifying-signed-events) page.

{% tabs %}
{% tab title="Python 3.11" %}

```python
import base64
import hashlib
import jwt


def verify_request(body, token, secret):
    # Decode the token using signature secret.
    try:
        payload = jwt.decode(token, secret, algorithms=["HS256"])
    except Exception as err:
        print(err)
        return False

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(base64.b64decode(body.encode('ascii')))
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        print('Checksum Mismatch')
        return False

    return True


def main(request):
    # Extract necessary request information.
    token = request['__ow_headers']['x-dt-signature']
    body = request['__ow_body']
    secret = request['DT_SIGNATURE_SECRET']

    # Validate request origin and content integrity.
    if not verify_request(body, token, secret):
        return {'statusCode': 400}
    #
    # Further processing here.
    #

    return {'statusCode': 200}
```

{% endtab %}

{% tab title="Node.js 16" %}

```javascript
const crypto = require('crypto')
const jwt = require('jsonwebtoken')
  
function verifyRequest(body, token, secret) {
    // Decode the token using signature secret.
    let decoded
    try {
        decoded = jwt.verify(token, secret)
    } catch(err) {
        console.log(err)
        return false
    }
    
    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(JSON.stringify(body)).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }
    
    return true
}

function main(params) {
    // Extract necessary request information.
    let buff   = new Buffer.from(params['__ow_body'], 'base64')
    let body   = JSON.parse(buff)
    let token  = params['__ow_headers']['x-dt-signature']
    let secret = params['DT_SIGNATURE_SECRET']
    
    // Validate request origin and content integrity.
    if (verifyRequest(body, token, secret) === false) {
        return {statusCode: 400}
    }
    
    //
    // Further processing here.
    //

    return { statusCode: 200 }
}
```

{% endtab %}
{% endtabs %}

### Add Parameter

Parameters are default values stored safely on IBM's servers. Once a request is received, it is merged with the incoming dictionary. Add a new parameter with the following Name and Value.

* **Name:** `DT_SIGNATURE_SECRET`
* **Value:** A unique password. We will use it later, so write it down.

### Configure Endpoint

Under the Endpoints tab, toggle **Enable as Web Action** and **Raw HTTP handling**. Copy the **HTTP Method URL** and save it for later as we will use it when configuring our Data Connector.

## Data Connector

To continuously forward the data to our newly created Cloud Action, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our [Creating a Data Connector](/data-connectors/creating-a-data-connector) guide. The following configurations should be set.

* **Endpoint URL:** The **HTTP Method URL** found in the previous step.
* **Signature Secret:** The value of **DT\_SIGNATURE\_SECRET** parameter set earlier.

![](/files/-MRto5TBuJfY3eA2vB6_)

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.

## Test the Integration <a href="#test-the-integration" id="test-the-integration"></a>

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each [Periodic Heartbeat](https://app.gitbook.com/@disruptive/s/developer-docs/~/drafts/-MSSn8rMbsoOFUC9VGeL/references/events#periodic-heartbeat) or by touching a sensor to force a new event.

![](/files/-MSDNrkr4EHaUlFAWn6A)

If instead the Error counter increments, a response containing a non-200 status code is returned.

* Verify that the Data Connector endpoint URL is correct.
* IBM Cloud provides a host of tools that can be used to [monitor Functions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-logs). Check the logs for any tracebacks that could explain why an error is returned.

## Next steps

Your sensor data is now in the IBM Cloud environment, and you can start using it in their various services. Fortunately, IBM has some well-documented guides to get you started.

### PostgreSQL Database

A database should be tailored to each specific use case. However, if you're uncertain, PostgreSQL (Postgres) is a good place to get started. The following guides will show you how to create a new Postgres database, then connect your Cloud Action to execute queries.

* [Getting Started with a PostgreSQL Database](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started).
* [Connecting an External Application](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-external-app).


# Development Guides

A few short guides on how to develop with Data Connectors.

## Overview

These guides are meant to help you get started with various integrations or tasks related to Data Connector development. To get you up and running quickly they do not go into details of third-party tools used, and where possible, refers to both internal and external guides as necessary.

* [Local Development with ngrok](/data-connectors/development-guides/local-development-with-ngrok)


# Local Development with ngrok

Develop cloud services locally by continuously forwarding real data from a Data Connector using ngrok.

## Overview

This guide looks at how ngrok can be used to create a local development environment for Data Connectors applications. In summary, the Data Connector is configured to forward events to an URL generated by ngrok, a free proxy that points all incoming traffic to a specified port on localhost where we can receive the events and quickly develop and debug our application,

## Prerequisites

* **Service Account**\
  Your User or Service Account must have the [role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Developer or higher.
* **Local Development**\
  It is assumed that you are familiar with the bare minimums of local development, like using a shell.

## ngrok

[ngrok](https://ngrok.com) is a free proxy service that enables forwarding HTTPS traffic from a publicly accessible URL to a port on localhost.&#x20;

### Installation

Most package managers should contain ngrok. If not, see the [official download page](https://ngrok.com/download).

* **Debian:** `apt install ngrok`
* **Arch:** [ngrok AUR](https://aur.archlinux.org/packages/ngrok/)
* **Mac:** `brew install --cask ngrok`
* **Windows:** The ngrok client binary can be found [here](https://ngrok.com/download).

### Starting the Proxy

In your shell of choice, run the `ngrok` command with the `http` argument, followed by the port to which you want to forward incoming traffic.

```bash
ngrok http 3000 
```

Once the proxy has started, all traffic directed to the randomly generated HTTPS URL will be directed to the specified port on your localhost.

```
ngrok by @inconshreveable                                                            (Ctrl+C to quit)

Session Status                online
Session Expires               1 hour, 59 minutes
Version                       2.3.35
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://5a880278718b.ngrok.io -> http://localhost:3000
Forwarding                    https://5a880278718b.ngrok.io -> http://localhost:3000

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00
```

If you're using the free version of ngrok, your session will expire after two hours. To restart, rerun the command. Note that this changes the URL.

### Localhost Path

All modifications to the generated URL path is reflected on the localhost. This can be useful to know as some local development frameworks, such as the one used by Azure Functions, utilizes pathing.

* **Original**\
  `https://5a880278718b.ngrok.io -> http://localhost:3000`<br>
* **Added path**\
  `https://5a880278718b.ngrok.io/some/path -> http://localhost:3000/some/path`

## Data Connector

To continuously forward events to localhost, [create a new Data Connector](/data-connectors/creating-a-data-connector) or use an existing one, setting the following configurations.

* **Endpoint URL**\
  The generated **HTTPS URL** of the form `https://5a880278718b.ngrok.io`  as provided by ngrok.\
  Note that the **HTTP URL** will not be accepted by the Data Connector.

## Local Server App

When forwarding Data Connector events to localhost using ngrok, if no response is given to the request, an `502 Bad Gateway` error will the thrown.

The following snippets implement a simple local server app that receives the Data Connector request, prints the content, and responds with a success message.

{% tabs %}
{% tab title="Python 3.11 - Flask" %}
Install the necessary dependencies.

```
pip install flask==2.3.2
```

Copy the following snippet to a local file `app.py`.

{% code title="app.py" %}

```python
from flask import Flask, request

app = Flask(__name__)


@app.route('/', methods=['POST'])
def print_request_contents():
    print(f'\nHeaders\n-------\n{request.headers}')
    print(f'Body\n----\n{request.get_json()}')
    return 'Success'
```

{% endcode %}

Start the local server by running `flask run --port 3000` in your shell.
{% endtab %}

{% tab title="Node.js 20 - Express" %}
Install the necessary dependencies.

```
npm install express@4.18.2 body-parser@1.20.2
```

Copy the following snippet to a local file `index.js`.

{% code title="index.js" %}

```javascript
const express = require("express")     // npm install express@4.18.1
const bodyParser = require("body-parser") // npm install body-parser@1.20.0

const app = express()
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())

app.post("/", (req, res) => {
    console.dir(req.body, { depth: null })
    res.send("Success")
})

app.listen(3000)
```

{% endcode %}

Start the local server by running `node index.js` in your shell.
{% endtab %}
{% endtabs %}

## Sensor Emulator

Our sensors emit events every [periodic heartbeat](/concepts/events#periodic-heartbeat) or when touched. When developing an application locally, being able to emit events at will can be quite useful. Our Sensor Emulator lets you create an emulated device for which events can be sent using a click of a button in DT Studio or the REST API.

You can read about how to create and emit emulated events on our [Sensor Emulator](/sensor-emulator) page.


# Introduction to REST API

A quick introduction to our REST API and how to get started using it.

## Overview

Our REST API can be used to interact with our services programmatically. For instance, DT Studio, our easy-to-use device management web app, is itself entirely built upon the REST API. Therefore, everything you can do in DT Studio, from moving devices between projects and managing Service Accounts, can also be done via our REST API.

## First Steps

The only thing you need to get started using our REST API is a Service Account with sufficient permissions for the task you want to do. While subsequent Service Accounts can be managed through the REST API itself, the initial account must be set up using DT Studio as an entry point.

Read our guide on [Creating a Service Account](/service-accounts/creating-a-service-account) to get started.

## Start Using our REST API

Once in possession of a Service Account, a comprehensive list of every endpoint available can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api) documentation.

### Shorthand Base URL

All our REST API calls are extensions of a base URL used in either long- or short-form.

* `https://api.d21s.com/v2`
* `https://api.disruptive-technologies.com/v2`

This is also true for our [Sensor Emulator](/sensor-emulator) API, an extension for the main REST API under a separate URL.

* `https://emulator.d21s.com/v2`
* `https://emulator.disruptive-technologies.com/v2`

### Authentication

Our REST API supports two methods of authentication, both using a Service Account for access control.

* **Basic Auth:** A simple username- and password-style authentication, handy for quick prototyping.
* **OAuth2 Flow:** A robust exchange of credentials for an access token used to authenticate.

We recommend using the OAuth2 authentication flow when integrating with our REST API. Once implemented, OAuth2 makes sending HTTP Requests both simple and secure.

Head over to our [REST API Authentication](/authentication) section to get started.


# Explore Our Endpoints

Many different tools and programming languages can be used to interact with our REST API. The following short guides are meant to help you get started with some popular ones. Once set up, every endpoint available in our [REST API Reference](http://developer.d21s.com/api) can be interacted with at will.

* [cURL](/explore-our-rest-api/with-curl)
* [Python API](https://developer.disruptive-technologies.com/api/libraries/python/index.html)
* [Postman](/explore-our-rest-api/with-postman)


# with cURL

A quick showcase of how cURL can be used to explore the various calls available on our REST API.

## Overview

We will here show how cURL can be used to interact with our REST API. As an example, Basic Auth will be used to authenticate a GET request that fetches a list of available projects. All available endpoints are listed in our [REST API Reference](https://developer.disruptive-technologies.com/api).

## Prerequisites

* **Enable Basic Auth**\
  You need to have [created a Service Account](/service-accounts/creating-a-service-account) with Basic Auth enabled. Any role will suffice.

## cURL

cURL is a command-line tool that can be used to easily make arbitrary HTTP requests. This includes our entire API which can be accessed via a single cURL command. Detailed information about the parameters used can be found under cURLs manpage, `man curl`.

### Installation

The tool comes preinstalled on most Linux and Mac systems. You can verify the install by opening your favorite shell and typing `which curl` , and are otherwise readily available in most package managers.

* **Debian-based distributions:** `apt install curl`
* **Arch-based distributions:** `pacman -S curl`
* **Mac:** `brew install curl`
* **Windows:** The cURL client binary can be found [here](https://curl.haxx.se/download.html#Win64).

## GET Request

The following command will fetch a list of all available projects for a Service Account.

```bash
curl -X GET "https://api.disruptive-technologies.com/v2/projects" \
    -u "<YOUR_SERVICE_ACCOUNT_KEY_ID>":"<YOUR_SERVICE_ACCOUNT_SECRET>"
```

* **-X \<command>**\
  Specifies the HTTP request method and to which URL it is sent.\
  The method, usually GET or POST, should be changed to accommodate the target URL.
* **-u \<user:password>**\
  Specifies the user for authentication.\
  We use a Service Account, and the key-id and secret can be found when creating the account, as explained in the [Basics of Service Accounts](/service-accounts/introduction-to-service-accounts).&#x20;

If the valid key-id and secret of a Service Account with sufficient privileges were used, cURL should return a list of the available projects for that account to stdout.

```javascript
{"projects":[{"name":"projects/btadoe94jplfdvvdr6q0","displayName":"wCaV7Y",
"organization":"organizations/b9spme0i0da0037ge49g","organizationDisplayName":
"Disruptive Technologies Research","sensorCount":10,"cloudConnectorCount":1,
"inventory":false}],"nextPageToken":""}
```

## POST Request

The following command adds the label `my-new-label` with value `some-value` to the specified device.

```bash
curl -X POST "https://api.d21s.com/v2/projects/<YOUR_PROJECT_ID>/devices/<YOUR_DEVICE_ID>/labels" \
    -u "<YOUR_SERVICE_ACCOUNT_KEY_ID>":"<YOUR_SERVICE_ACCOUNT_SECRET>" \
    -d '{"key": "my-new-label", "value": "some-value"}'
```

* **-X \<command>**\
  Specifies the HTTP request method and to which URL it is sent.\
  The method, usually GET or POST, should be changed to accommodate the target URL.
* **-u \<user:password>**\
  Specifies the user for authentication.\
  We use a Service Account, and the key-id and secret can be found when creating the account, as explained in the [Basics of Service Accounts](/service-accounts/introduction-to-service-accounts).&#x20;
* **-d \<data>**\
  Sends the specified data in a POST request to the HTTP server.

If the valid key-id and secret of a Service Account with sufficient privileges were used, cURL should return a list of the available projects for that account to stdout.

```javascript
{
  "name": "projects/<YOUR_PROJECT_ID>/devices/<YOUR_DEVICE_ID>/labels/my-new-label",
  "key": "my-new-label",
  "value": "some-value"
}
```

## Optional Formatting

cURL alone makes no attempt at formatting its output. By piping the cURL output through a command-line JSON processor like [jq](https://stedolan.github.io/jq/),  the result is much more readable.&#x20;

```bash
curl <insert-command-here> | jq
```

The same response as previously now looks like the following.

```bash
{
  "projects": [
    {
      "name": "projects/btadoe94jplfdvvdr6q0",
      "displayName": "wCaV7Y",
      "organization": "organizations/b9spme0i0da0037ge49g",
      "organizationDisplayName": "Disruptive Technologies Research",
      "sensorCount": 10,
      "cloudConnectorCount": 1,
      "inventory": false
    }
  ],
  "nextPageToken": ""
}
```


# with Postman

A quick showcase of how Postman can be used to explore the various calls available on our REST API.

## Overview

We will here show how Postman can be used to interact with our REST API. As an example, Basic Auth will be used to authenticate a GET request that fetches a list of available projects and a POST request that creates a new label for a device. All available endpoints are listed in our [REST API Reference](https://developer.disruptive-technologies.com/api).

## Prerequisites

* **Enable Basic Auth**\
  You need to have [created a Service Account](/service-accounts/creating-a-service-account) with Basic Auth and a role of Project Developer or higher.

## Postman

Postman is a collaboration platform for API development. It is a powerful tool with many useful features, but we will here use it to make simple arbitrary HTTP requests.

### Installation

Postman can be acquired from the official [downloads page](https://www.postman.com/downloads/).

### Authorization

When creating a new request, under the **Authorization** tab, specify the method of Basic Auth.

* **Username:** Your Service Account Key ID.
* **Password:** Your Service Account Secret.

![](/files/-MTaLVnvrtZqiJe0rC7s)

## GET Request

To fetch a list of all projects available to a Service Account, create a new GET request with the following endpoint URL. No request body is required for this call.

`https://api.d21s.com/v2/projects`

![](/files/-MTaSkp7jNUND1UcApb7)

If the valid key-id and secret of a Service Account with sufficient privileges were used, Postman should return a list of the available project for that account.

```javascript
{
    "projects": [
        {
            "name": "projects/<PROJECT_ID>",
            "displayName": "wCaV7Y",
            "organization": "organizations/<ORGANIZATION_ID>",
            "organizationDisplayName": "Disruptive Technologies Research",
            "sensorCount": 10,
            "cloudConnectorCount": 1,
            "inventory": false
        }
    ],
    "nextPageToken": ""
}
```

## POST Request

To add a new label to a device, create a new POST request with the following endpoint URL.

`https://api.d21s.com/v2/projects/<PROJECT_ID>/devices/<DEVICE_ID>/labels`

For the body, choose Raw and then JSON. Within, the label name and value should be provided.

![](/files/-MTaSKgKTAU2y2UmADNs)

If the valid key-id and secret of a Service Account with sufficient privileges were used, Postman should return the name, key, and value.

```javascript
{
    "name": "projects/<PROJECT_ID>/devices/<DEVICE_ID>/labels/my-new-label",
    "key": "my-new-label",
    "value": "some-value"
}
```


# Authentication

An overview of the two supported methods of authenticating our REST API.

## OAuth2&#x20;

We recommend using the OAuth2 authentication flow when integrating with our REST API. It utilizes the [JWT](https://jwt.io/) as a medium of exchange and a [Service Account](/service-accounts/introduction-to-service-accounts) for access control. After the authentication exchange is implemented, the returned access token allows you to perform HTTP requests efficiently.

Read about how to implement [OAuth2 REST API Authentication](/authentication/oauth2) to get started.

## Basic Auth

We also support Basic Auth for authenticating with a Service Account. While this is not recommended for production-level integrations, it can be quite useful for quick experimentation and prototyping. It is disabled by default but can be [enabled as desired](/service-accounts/creating-a-service-account#create-new).

Read about how to implement [Basic Auth](/authentication/basic-auth) to get started.


# OAuth2

A guide on how to implement an OAuth2 flow for authenticating our REST API.

## Overview

This guide aims to provide the basic understanding and ability to authenticate a Service Account for REST API integrations. Core concepts about the OAuth2 authentication flow are briefly explained, and an example implementation is provided. The implementation is tested using the returned access token to send a request to the REST API to list all available projects.

## Prerequisites

* **Service Account Credentials**\
  A [Service Account](/service-accounts/creating-a-service-account) must be created with a membership in the target project. Any role will suffice.

## Example Code

The provided example code can be summarized by the following points.

* A JSON Web Token (JWT) is constructed and signed with a secret.
* The JWT is exchanged for an access token.
* The Access token is used to authenticate the REST API.

The implemented OAuth2 authentication flow is based on the [RFC7523 specification](https://tools.ietf.org/html/rfc7523).

### Environment Setup

If you wish to run the code locally, make sure you have a working runtime environment.

{% tabs %}
{% tab title="Python 3.11" %}
The following packages are required by the example code and must be installed.

```
pyjwt==2.7.0
requests==2.31.0
```

{% endtab %}

{% tab title="Python API" %}
The [Disruptive Technologies Python API](https://developer.disruptive-technologies.com/api/libraries/python/client/quickstart.html#client-quickstart) can be installed through pip.

```
pip install --upgrade disruptive
```

{% endtab %}

{% tab title="Node.js 20" %}
The following modules are required by the example code and must be installed.

```bash
npm install jsonwebtoken@9.0.1
npm install axios@1.4.0
```

{% endtab %}

{% tab title="Go 1.20" %}
The following packages are required by the example code and must be installed.

```
go get -u github.com/golang-jwt/jwt/v5@v5.0.0
```

{% endtab %}
{% endtabs %}

Add the following environment variables as they will be used in the authentication flow.

{% tabs %}
{% tab title="Bash" %}

```bash
export DT_SERVICE_ACCOUNT_KEY_ID=<YOUR_SERVICE_ACCOUNT_KEY_ID>
export DT_SERVICE_ACCOUNT_SECRET=<YOUR_SERVICE_ACCOUNT_SECRET>
export DT_SERVICE_ACCOUNT_EMAIL=<YOUR_SERVICE_ACCOUNT_EMAIL>
```

{% endtab %}
{% endtabs %}

### Source

{% tabs %}
{% tab title="Python 3.11" %}

```python
import os
import time

import jwt
import requests

# Authentication details used in the OAuth2 flow.
SERVICE_ACCOUNT_KEY_ID = os.getenv('DT_SERVICE_ACCOUNT_KEY_ID', '')
SERVICE_ACCOUNT_SECRET = os.getenv('DT_SERVICE_ACCOUNT_SECRET', '')
SERVICE_ACCOUNT_EMAIL = os.getenv('DT_SERVICE_ACCOUNT_EMAIL', '')


def get_access_token(key_id, email, secret):
    # Verify none of the credentials are missing.
    for variable in [key_id, email, secret]:
        if len(variable) < 1:
            raise ValueError('One or more credentials not set.')

    # Construct the JWT header.
    jwt_headers = {
        'alg': 'HS256',
        'kid': key_id,
    }

    # Construct the JWT payload.
    now = time.time()
    jwt_payload = {
        'iat': int(now),         # current time in unixtime
        'exp': int(now) + 3600,  # expiration time in unixtime
        'aud': 'https://identity.disruptive-technologies.com/oauth2/token',
        'iss': email,
    }

    # Sign and encode JWT with the secret.
    encoded_jwt = jwt.encode(
        payload=jwt_payload,
        key=secret,
        algorithm='HS256',
        headers=jwt_headers,
    )

    # Prepare HTTP POST request data.
    # note: The requests package applies Form URL-Encoding by default.
    request_data = {
        'assertion': encoded_jwt,
        'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer'
    }

    # Exchange the JWT for an access token.
    access_token_response = requests.post(
        url='https://identity.disruptive-technologies.com/oauth2/token',
        headers={'Content-Type': 'application/x-www-form-urlencoded'},
        data=request_data,
    )

    # Halt if response contains an error.
    if access_token_response.status_code != 200:
        print('Status Code: {}'.format(access_token_response.status_code))
        print(access_token_response.json())
        return None

    # Return the access token in the request.
    return access_token_response.json()['access_token']


def main():
    # Get an access token using an OAuth2 authentication flow.
    access_token = get_access_token(
        SERVICE_ACCOUNT_KEY_ID,
        SERVICE_ACCOUNT_EMAIL,
        SERVICE_ACCOUNT_SECRET,
    )

    # Verify that we got a valid token back.
    if access_token is None:
        return

    # Test the token by sending a GET request for a list of projects.
    print(requests.get(
        url='https://api.disruptive-technologies.com/v2/projects',
        headers={'Authorization': 'Bearer ' + access_token},
    ).json())


if __name__ == '__main__':
    main()
```

{% endtab %}

{% tab title="Python API" %}
Our Python API implements the OAuth2 flow for you, requiring no further action.

```python
import os

import disruptive as dt

dt.default_auth = dt.Auth.service_account(
    key_id=os.getenv('DT_SERVICE_ACCOUNT_KEY_ID', ''),
    secret=os.getenv('DT_SERVICE_ACCOUNT_SECRET', ''),
    email=os.getenv('DT_SERVICE_ACCOUNT_EMAIL', ''),
)
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
const jwt = require('jsonwebtoken')
const axios = require('axios').default

// Authentication details used in the OAuth2 flow.
const serviceAccountKeyID = process.env.DT_SERVICE_ACCOUNT_KEY_ID
const serviceAccountSecret = process.env.DT_SERVICE_ACCOUNT_SECRET
const serviceAccountEmail = process.env.DT_SERVICE_ACCOUNT_EMAIL

// Creates a JWT from the arguments, and exchanges it for an 
// access token which is returned as a promise. If an error 
// occurs at any point, an error is thrown and the returned 
// promise is rejected.
async function getAccessToken(keyID, email, secret) {
    let credentials = [keyID, email, secret]
    for (let i = 0; i < credentials.length; i++) {
        if (credentials[i] === undefined) {
            throw new Error("One or more variables are missing.")
        }
    }

    // Construct the JWT header.
    let jwtHeaders = {
        'alg': 'HS256',
        'kid': keyID,
    }

    // Construct the JWT payload.
    const now = Date.now()
    let jwtPayload = {
        'iat': Math.floor(now / 1000),        // current time in  unixtime
        'exp': Math.floor(now / 1000) + 3600, // expiration time in unixtime
        'aud': 'https://identity.disruptive-technologies.com/oauth2/token',
        'iss': email,
    }

    // Sign and encode JWT with the secret.
    const jwtEncoded = jwt.sign(
        jwtPayload,
        secret,
        {
            header: jwtHeaders,
            algorithm: 'HS256',
        },
    )

    // Prepare POST request data.
    const requestObject = {
        'assertion': jwtEncoded,
        'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer',
    }

    // Converts the requestObject to a Form URL-Encoded string.
    const requestData = Object.keys(requestObject).map(function(key) {
        return encodeURIComponent(key) + '=' + encodeURIComponent(requestObject[key])
    }).join('&')

    // Exchange JWT for access token.
    const accessTokenResponse = await axios({
        method: 'POST',
        url: 'https://identity.disruptive-technologies.com/oauth2/token',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
        data: requestData,
    }).catch(function(error) {
        // Prints the error response (if any), an re-throws the error.
        if (error.response) {
            console.log(error.response.data)
        }
        throw error
    })

    // Return the access token in the request.
    return accessTokenResponse.data.access_token
}
async function main() {
    // Get an access token using an OAuth2 authentication flow.
    const accessToken = await getAccessToken(
        serviceAccountKeyID,
        serviceAccountEmail,
        serviceAccountSecret,
    )

    // Test the token by sending a GET request for a list of projects.
    const response = await axios({
        method: 'GET',
        url: 'https://api.disruptive-technologies.com/v2/projects',
        headers: { 'Authorization': 'Bearer ' + accessToken },
    })

    // Print response data.
    console.log(JSON.stringify(response.data, null, 2))
}
main()
```

{% endtab %}

{% tab title="Go 1.20" %}

```go
package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"os"
	"strings"
	"time"

	jwt "github.com/golang-jwt/jwt/v5"
)

const (
	// Used to exchange a JWT for an access token.
	tokenEndpoint = "https://identity.disruptive-technologies.com/oauth2/token"
	// Base URL for the Disruptive REST API.
	apiBaseUrl = "https://api.disruptive-technologies.com/v2"
)

type AuthResponse struct {
	// The access token used to access the Disruptive REST API.
	AccessToken string `json:"access_token"`
	// The type of token this is. Will typically be "Bearer".
	TokenType string `json:"token_type"`
	// How many seconds until the token expires. Will typically be 3600.
	ExpiresIn int `json:"expires_in"`
}

func getAccessToken(keyID string, secret string, email string) (*AuthResponse, error) {
	// Check that none of the credentials are missing.
	var credentials = []string{keyID, secret, email}
	for _, value := range credentials {
		if value == "" {
			return nil, errors.New("One or more credentials missing")
		}
	}

	// Construct the JWT header.
	jwtHeader := map[string]interface{}{
		"alg": "HS256",
		"kid": keyID,
	}

	// Construct the JWT payload.
	now := time.Now()
	jwtPayload := &jwt.RegisteredClaims{
		Issuer:    email,
		Audience:  jwt.ClaimStrings{tokenEndpoint},
		IssuedAt:  jwt.NewNumericDate(now),
		ExpiresAt: jwt.NewNumericDate(now.Add(time.Hour)),
	}

	// Sign and encode JWT with the secret.
	token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwtPayload)
	token.Header = jwtHeader
	encodedJwt, _ := token.SignedString([]byte(secret))

	// Prepare HTTP POST request data.
	// NOTE: The body must be Form URL-Encoded.
	reqData := url.Values{
		"assertion":  {encodedJwt},
		"grant_type": {"urn:ietf:params:oauth:grant-type:jwt-bearer"},
	}.Encode()

	// Create the request to exchange the JWT for an access token.
	req, err := http.NewRequest(
		"POST",
		tokenEndpoint,
		strings.NewReader(reqData),
	)
	if err != nil {
		return nil, err
	}

	// Set Content-Type header to specify that our body is Form-URL Encoded.
	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

	// Exchange the JWT for an access token. Set a 3 second
	// timeout in case the server can't be reached.
	client := &http.Client{Timeout: time.Second * 3}
	res, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer res.Body.Close()

	// Decode the response body to an AuthResponse.
	var authResponse AuthResponse
	if err := json.NewDecoder(res.Body).Decode(&authResponse); err != nil {
		return nil, err
	}

	// Return the AuthResponse, which contains the access token.
	return &authResponse, nil
}
func listProjects(auth *AuthResponse) error {
	// Create the request to get a list of projects from the Disruptive REST API
	req, err := http.NewRequest("GET", apiBaseUrl+"/projects", nil)
	if err != nil {
		return err
	}

	// Set the Authorization header by specifying both the token type
	// (which will typically be "Bearer") as well as the access token
	req.Header.Set("Authorization", fmt.Sprintf("%s %s", auth.TokenType, auth.AccessToken))

	// Create an http Client with a timeout
	// Send the GET request to list all projects.
	// Set a 3 second timeout in case the server can't be reached.
	client := &http.Client{Timeout: time.Second * 3}
	resp, err := client.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	// Define a struct with the format we expect the response
	// to be in. See the REST API Reference for more details.
	type ProjectsResponse struct {
		Projects []struct {
			Name                    string `json:"name"`
			DisplayName             string `json:"displayName"`
			Inventory               bool   `json:"inventory"`
			Organization            string `json:"organization"`
			OrganizationDisplayName string `json:"organizationDisplayName"`
			SensorCount             int    `json:"sensorCount"`
			CloudConnectorCount     int    `json:"cloudConnectorCount"`
		}
		NextPageToken string `json:"nextPageToken"`
	}

	// Decode the response into a ProjectsResponse.
	var projectsResponse ProjectsResponse
	if err = json.NewDecoder(resp.Body).Decode(&projectsResponse); err != nil {
		return err
	}

	// Print the name of each project, and how many device each contain.
	for _, project := range projectsResponse.Projects {
		fmt.Println(project.DisplayName)
		fmt.Printf("  %d Sensors\n", project.SensorCount)
		fmt.Printf("  %d Cloud Connectors\n", project.CloudConnectorCount)
	}

	return nil
}

func main() {
	// OAuth2 authentication flow.
	auth, err := getAccessToken(
		os.Getenv("DT_SERVICE_ACCOUNT_KEY_ID"),
		os.Getenv("DT_SERVICE_ACCOUNT_SECRET"),
		os.Getenv("DT_SERVICE_ACCOUNT_EMAIL"),
	)
	if err != nil {
		log.Fatal(err)
	}

	// Test the access token by listing all the projects our
	// Service Account has access to.
	if err := listProjects(auth); err != nil {
		log.Fatal(err)
	}
}
```

{% endtab %}
{% endtabs %}

## Authentication Flow

Authenticating a client is a 3 step process, as illustrated below. Using both the figure and example code, the following sections will explain in more detail what each step entails.

{% tabs %}
{% tab title="Create the JWT" %}
![](/files/-MTfBd8c4p1X5vGKt4hY)
{% endtab %}

{% tab title="Exchange for Access Token" %}
![](/files/-MTfBh9hYsG9v9UiJKkf)
{% endtab %}

{% tab title="Access the REST API" %}
![](/files/-MTfBk82r_I9rQnxsEcS)
{% endtab %}
{% endtabs %}

### Create the JWT

A JSON Web Token (JWT) contains three fields:

1. **Header:** Token type and signature algorithm.
2. **Payload:** Claims and additional data.
3. **Signature:** A signature calculated of the entire JWT + a private secret.

Before being sent, these fields are each Base64Url encoded. They are combined in a compact dot format in the form **Base64Url(header).Base64Url(payload).Base64Url(signature)**, which is what we will refer to as the encoded JWT.

{% hint style="info" %}
&#x20;This guide will not cover much more details of JWT. For a great introduction on JWT that provides an interactive editor and an exhaustive list of client libraries, please see [jwt.io](https://jwt.io/).
{% endhint %}

Using your Service Account credentials, construct the JWT headers and payload. Here, `iat` is the issuing time, and `exp` the expiration time of a maximum 1 hour after `iat`.

{% tabs %}
{% tab title="Python 3.11" %}

```python
# Construct the JWT header.
jwt_headers = {
    'alg': 'HS256',
    'kid': key_id,
}

# Construct the JWT payload.
jwt_payload = {
    'iat': int(time.time()),        # current unixtime
    'exp': int(time.time()) + 3600, # expiration unixtime
    'aud': 'https://identity.disruptive-technologies.com/oauth2/token',
    'iss': email,
}
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
// Construct the JWT header.
let jwtHeaders = {
    'alg': 'HS256',
    'kid': keyID,
}

// Construct the JWT payload.
let jwtPayload = {
    'iat': Math.floor(Date.now() / 1000),        // current unixtime
    'exp': Math.floor(Date.now() / 1000) + 3600, // expiration unixtime
    'aud': 'https://identity.disruptive-technologies.com/oauth2/token',
    'iss': email,
}
```

{% endtab %}

{% tab title="Go 1.20" %}

```go
// Construct the JWT header
jwtHeader := map[string]interface{}{
	"alg": "HS256",
	"kid": keyID,
}

// Construct the JWT payload
now := time.Now()
jwtPayload := &jwt.RegisteredClaims{
	Issuer:    email,
	Audience:  jwt.ClaimStrings{tokenEndpoint},
	IssuedAt:  jwt.NewNumericDate(now),
	ExpiresAt: jwt.NewNumericDate(now.Add(time.Hour)),
}
```

{% endtab %}
{% endtabs %}

The simplest way of Base64-encoding and signing our JWT is to use some language-specific library. This is available in most languages but can be done manually if desired.

{% tabs %}
{% tab title="Python 3.11" %}

```python
# Sign and encode JWT with the secret.
encoded_jwt = jwt.encode(
    payload   = jwt_payload,
    key       = secret,
    algorithm = 'HS256',
    headers   = jwt_headers,
)
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
// Sign and encode JWT with the secret.
const jwtEncoded = jwt.sign(
    jwtPayload,
    secret,
    {
        header: jwtHeaders,
        algorithm: 'HS256',
    },
)
```

{% endtab %}

{% tab title="Go 1.20" %}

```go
// Sign and encode JWT with the secret
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwtPayload)
token.Header = jwtHeader
encodedJwt, _ := token.SignedString([]byte(secret))
```

{% endtab %}
{% endtabs %}

### Exchange for Access Token

The encoded JWT is exchanged for an Access Token by sending a POST request to the same endpoint used to construct the JWT, namely `https://identity.disruptive-technologies.com/oauth2/token`.

The POST request header should include a **Content-Type** field indicating the format of the body. Additionally, the POST request body is Form URL-Encoded and contains the following fields:

1. "**assertion**" - Contains the encoded JWT string.
2. "**grant\_type**" - Contains the string `"urn:ietf:params:oauth:grant-type:jwt-bearer"`. This specifies that you want to exchange a JWT for an Access Token.

It is important to note that the data **has to be Form URL-Encoded**. Like Python's [requests](https://docs.python-requests.org/en/master/), some libraries do this by default and require no further input by the user. This is, however, not the norm and likely requires an additional step before sending the request. The URL Form Encoded data should have the following format.

```
assertion=Base64Url(header).Base64Url(payload).Base64Url(signature)&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer
```

The header, payload, and signature are found in the previous step.

{% tabs %}
{% tab title="Python 3.11" %}

```python
# Prepare HTTP POST request data.
# Note: The requests package applies Form URL-Encoding by default.
request_data = {
    'assertion': encoded_jwt,
    'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer'
}

# Exchange the JWT for an access token.
access_token_response = requests.post(
    url = 'https://identity.disruptive-technologies.com/oauth2/token',
    headers = { 'Content-Type': 'application/x-www-form-urlencoded' },
    data = request_data,
)
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
// Prepare POST request data.
const requestObject = {
    'assertion': jwtEncoded,
    'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer',
}

// Converts the requestObject to a Form URL-Encoded string.
const requestData = Object.keys(requestObject).map(function(key) {
    return encodeURIComponent(key) + '=' + encodeURIComponent(requestObject[key])
}).join('&')

// Exchange JWT for access token.
const accessTokenResponse = await axios({
    method: 'POST',
    url: 'https://identity.disruptive-technologies.com/oauth2/token',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    data: requestData,
}).catch(function (error) {
    if (error.response) {
        console.log(error.response.data)
    }
    throw error
})
```

{% endtab %}

{% tab title="Go 1.20" %}

```go
// Prepare HTTP POST request data.
// NOTE: The body must be Form URL-Encoded
reqData := url.Values{
	"assertion":  {encodedJwt},
	"grant_type": {"urn:ietf:params:oauth:grant-type:jwt-bearer"},
}.Encode()

// Create the request to exchange the JWT for an access token
req, err := http.NewRequest(
	"POST",
	tokenEndpoint,
	strings.NewReader(reqData),
)
if err != nil {
	return nil, err
}

// Set Content-Type header to specify that our body
// is Form-URL Encoded
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

// Exchange the JWT for an access token. Set a 3 second
// timeout in case the server can't be reached.
client := &http.Client{Timeout: time.Second * 3}
res, err := client.Do(req)
if err != nil {
	return nil, err
}
defer res.Body.Close()
```

{% endtab %}
{% endtabs %}

The access token response can be expected to have the following format. **It will be valid for 1 hour only.** To get a new token, perform the previous steps again.

```javascript
{
    "access_token": "d663e83546294b158fea2574a1945319",
    "token_type": "bearer",
    "expires_in": 3599
}
```

Check the [Common Errors](#common-errors) section at the bottom to troubleshoot errors.

{% hint style="info" %}
**Cache your access token**

The access token should be cached so that you can re-use it for all subsequent API calls. This will improve the overall performance of your integration.

A new access token should only be created a short while before the access token expires (for example a minute before), or if you receive a 401 status code from the API, indicating that you have an invalid access token.
{% endhint %}

### Access the REST API

Once you have the Access Token, you need to include this with every call to the API. This can be achieved by including the **Authorization** header in the form shown in the snippet below.

{% tabs %}
{% tab title="Python 3.11" %}

```python
# Test the access token by sending a GET request to get a list of projects.
print(requests.get(
    url = 'https://api.disruptive-technologies.com/v2/projects',
    headers = {'Authorization': 'Bearer ' + access_token},
).json())
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
// Test the token by sending a GET request for a list of projects.
const response = await axios({
    method: 'GET',
    url: 'https://api.disruptive-technologies.com/v2/projects',
    headers: { 'Authorization': 'Bearer ' + accessToken },
})
```

{% endtab %}

{% tab title="Go 1.20" %}

```go
// Test the access token by listing all the projects our
// Service Account has access to
if err := listProjects(auth); err != nil {
	log.Fatal(err)
}
```

{% endtab %}
{% endtabs %}

## Common Errors

This section covers the most commonly encountered errors while setting up Service Account authentication using OAuth2.&#x20;

Note that all errors returned from the `oauth2/token` with an HTTP status code of 400 indicates that there is something incorrect in your implementation. The specific errors returned from this endpoint might change in the future, so you should avoid writing logic to try to figure out which type of error you got. Instead, you should log out the response body you got back to see what's wrong and update your implementation accordingly.

For an overview of all REST API errors, see the [Error Codes](/error-codes) reference page.

### **Timing Error**

```json
{
    "error": "invalid_grant", 
    "error_description": "Timing-related error. Check the 'exp' and 'iat' claims."
}
```

There is something incorrect with either the issue time (`iat`) or the expiration time (`exp`) claims in your JWT. Make sure that both claims are set, and have the correct values according to the documentation above.

Here are some typical reasons why you might receive this error:

* Either the `iat` or `exp` claims (or both) are missing in the JWT.
* The `iat` claim is too far in the past or future compared to "now".
* The `exp` claim is more than 3600 seconds past `iat` or before `iat`.
* The computer running the code does not have its clock set correctly. Make sure your computer is regularly doing [NTP syncs](https://en.wikipedia.org/wiki/Network_Time_Protocol), and check [time.is](https://time.is/) to verify that your computer's clock is set correctly.

### Untrusted Entity Error

<pre class="language-json"><code class="lang-json"><strong>{
</strong>    "error": "invalid_grant", 
    "error_description": "Untrusted entity. Check the 'aud' and 'iss' claims."
}
</code></pre>

This might be caused by one of the following reasons:

* Either the audience (`aud`) claim or issuer (`iss`) claim are missing.
* The `aud` claim is not set to `https://identity.disruptive-technologies.com/oauth2/token`.
* The `iss` claim is not set to the email of the Service Account.

### Signature Error

```json
{
    "error": "invalid_grant", 
    "error_description": "Invalid signature"
}
```

The JWT has not been signed correctly. Make sure you have used the correct secret to sign the JWT. It should be the secret of the Service Account Key that corresponds to the key id used in the `kid` field of the JWT header.

It's recommended to use a library to do the heavy lifting of signing the JWT for you. Check out [jwt.io](https://jwt.io/libraries) for a selection of recommended libraries in many different languages. If you're writing the JWT signature implementation yourself, check out [this guide](https://dev.to/kimmaida/signing-and-validating-json-web-tokens-jwt-for-everyone-25fb) for more information on how to do this.

### Unsupported Grant Type

```json
{
    "error": "unsupported_grant_type"
}
```

This might be caused by one of the following reasons:

* The `Content-Type` HTTP header is not set to `application/x-www-form-urlencoded`.
* The `grant_type` field is missing from the POST request body.
* The body of the POST request is not Form URL-encoded.

### Other

```json
{
    "error": "invalid_grant"
}
```

This might be caused by one of the following reasons:

* The Service Account doesn't exist.
* The Service Account key doesn't exist.
* The `assertion` field of the request (the encoded JWT) is missing or malformed. Make sure that the JWT is formatted correctly, and that the entire request body is Form URL-encoded as described in the documentation above.

### 403 Errors From REST API

If you get a 403 HTTP status code from the REST API when using the access token you just generated, it's likely not an issue with the authentication. The most common reason for this is that the Service Account you are authenticating as does not yet have access to the resource you're trying to access. Before a Service Account can use the API it needs to be a member of either a project or an organization. See the [Managing Access Rights](/service-accounts/managing-access-rights) page for more information about how to add a Service Account as a member. The [Organizational Structures](/service-accounts/organizational-structures) page gives some guidance for when to add Service Accounts as members on a project vs. on an organization.

## Next Steps

You are now familiar with the OAuth2 authentication routine, but it must still be refreshed every hour for integrations that run continuously. For this, see our example on [Refreshing Access Token](/examples/refreshing-access-token).


# Basic Auth

A guide on how to use Basic Auth for authenticating our REST API.

## Overview

Basic Auth is supported in most request libraries and is often as simple as adding a username- and password parameter. To get you up and running quickly, we present a few language-specific methods by fetching a list of projects available from the REST API using a Service Account for access control.

## Prerequisites

* **Service Account Credentials**\
  A [Service Account](/service-accounts/creating-a-service-account) must be created with a membership in the target project. Any role will suffice.

## Code Sample

Add the following environment variables as they will be used to authenticate the API. Replace the values of the form `<VARIABLE>` with your own details.

```bash
export DT_SERVICE_ACCOUNT_KEY_ID=<YOUR_SERVICE_ACCOUNT_KEY_ID>
export DT_SERVICE_ACCOUNT_SECRET=<YOUR_SERVICE_ACCOUNT_SECRET>
```

Create and enter a new directory that will contain the example project.

{% tabs %}
{% tab title="Python 3.11" %}
Install the necessary dependencies.

```bash
requests==2.31.0
```

Create a file `main.py` with the following content.

```python
import os
import requests

if __name__ == '__main__':
    projects = requests.get(
        url='https://api.d21s.com/v2/projects',
        auth=(
            os.environ.get('DT_SERVICE_ACCOUNT_KEY_ID'),
            os.environ.get('DT_SERVICE_ACCOUNT_SECRET'),
        )
    )

    print(projects.json())
```

{% endtab %}

{% tab title="Node.js 20" %}
Install the necessary dependencies.

```bash
npm install axios@1.4.0
```

Create a file `index.js` with the following content.

```javascript
const axios = require('axios').default;

async function main() {
    const response = await axios({
        method: 'GET',
        url: 'https://api.disruptive-technologies.com/v2/projects',
        auth: {
            username: process.env.DT_SERVICE_ACCOUNT_KEY_ID,
            password: process.env.DT_SERVICE_ACCOUNT_SECRET,
        }
    })

    console.log(JSON.stringify(response.data, null, 2))
}
main()
```

{% endtab %}

{% tab title="Go 1.20" %}
Initialize a new Go project.

```bash
go mod init example
```

Create a file `main.go` with the following content.

```go
package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"os"
	"time"
)

func main() {
	// Construct endpoint URL for listing devices in project.
	deviceListUrl := "https://api.d21s.com/v2/projects"

	// Create a custom http Client with timeout.
	client := &http.Client{Timeout: time.Second * 3}

	// Create the request object with method, URL, but no optional body.
	req, err := http.NewRequest("GET", deviceListUrl, nil)
	if err != nil {
		log.Fatal(err)
	}

	// Set the request's Authorization header to use HTTP Basic Authentication.
	req.SetBasicAuth(
		os.Getenv("DT_SERVICE_ACCOUNT_KEY_ID"),
		os.Getenv("DT_SERVICE_ACCOUNT_SECRET"),
	)

	// Send an HTTP request and return an HTTP response.
	response, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer response.Body.Close()

	// Convert response body to map.
	var body map[string]interface{}
	if err = json.NewDecoder(response.Body).Decode(&body); err != nil {
		log.Fatal(err)
	}

	// Pretty print the response body.
	prettyBody, _ := json.MarshalIndent(body, "", "    ")
	fmt.Println(string(prettyBody))
}

```

{% endtab %}

{% tab title="cURL" %}

```bash
curl -X GET "https://api.disruptive-technologies.com/v2/projects" \
    -H "accept: application/json" \
    -u $DT_SERVICE_ACCOUNT_KEY_ID:$DT_SERVICE_ACCOUNT_SECRET
```

{% endtab %}
{% endtabs %}

Running the file should list all projects available to the Service Account.

While this example sends a GET request to the API that lists all available projects, the request URL can be replaced with any call in our [REST API Reference](https://developer.disruptive-technologies.com/api).


# Error Codes

A summary of errors that can be encountered when using our REST API.

## Response Body

The response body contains information about the error and typically contains the following three fields, but this may vary depending on the error. An example error response body is shown below.

```javascript
{
    "error": "already exists",
    "code": 409,
    "help": "https://docs.d21s.com/error/#409"
}
```

| Field   | Type     | Description                                    |
| ------- | -------- | ---------------------------------------------- |
| `error` | `string` | A summary of what went wrong.                  |
| `code`  | `int`    | HTTP status code of the response.              |
| `help`  | `string` | Developer documentation link for more details. |

## 400 - Bad Request <a href="#id-400" id="id-400"></a>

This is most likely due to an invalid argument as a part of a path, query parameter, or request body. Please see the error message for more information.

## 401 - Unauthorized <a href="#id-401" id="id-401"></a>

The user or the Service Account can not be authenticated towards our API.

#### Access Tokens

If you are using an Access Token received from our OAuth2 endpoint, please note that these are only valid for *one hour*. For uninterrupted operations using Access Tokens, make sure to get a new one before the previous one expires.&#x20;

See our [Authentication guide](/authentication/oauth2) for more details.

## 403 - Forbidden <a href="#id-403" id="id-403"></a>

The user or Service Account does not have access to this resource. Make sure you have a sufficiently elevated role. A list of permissions granted by each role can be found on the [Permissions](/service-accounts/permissions) page.

## 404 - Not Found <a href="#id-404" id="id-404"></a>

The resource that you are looking for doesn’t exist.

When executing a method in the form of`/projects/{PROJECT_ID}/devices:transfer`, where the last argument is separated by a `:`, this error indicates that the  PROJECT\_ID cannot be found, not that the devices slated for transfer were not found.

#### Propagation Delay

When creating a resource using our REST API, a few seconds will pass before the resource is available. To avoid a 404 response, implement a short delay of 1 second before requesting the resource.

## 409 - Conflict <a href="#id-409" id="id-409"></a>

The resource that you are trying to create most likely already exists.

You will get this error if you are trying to create a new label with the same name as an existing one, or if you invite a member to an organization or project who is already invited.

It is also possible to get this error if you are trying to change a resource rapidly from different clients at once. If you suspect this, then read out the related resources again, apply any changes, and try to do the operation again.

## 429 - Too many requests <a href="#id-429" id="id-429"></a>

You are sending too many requests to the API too quickly.

When you exceed your rate limit of 75 requests per 10 seconds, you will see this error. This represents a steady state of 7.5 requests per second, but we also allow for shorter bursts, up to the limit of 75 requests per 10 seconds.

#### Retry-After header

When you exceed this limit, this error response will have an attached `Retry-After` header. This header can be used to schedule a retry no earlier than the provided `Retry-After` value in seconds.&#x20;

For example, let us say that you are trying to read-modify-write a specific label on many devices as quickly as possible. After running 75 requests in 1 second, you will get a “429 - Too many requests” error response. In response, you will get a `Retry-After` which will tell you to wait 9 seconds before retrying again.

Please note that while this burst-wait-burst is possible, spreading out requests over time is preferred.

## 500 - Internal server error <a href="#id-500" id="id-500"></a>

There has been an internal error in our services.

This error is usually intermittent, and we recommend you retry the request with an exponential backoff with a minimum 1-second delay.

If the error persists, please check our [Status Page](https://status.disruptive-technologies.com/). If there is no announced downtime, please get in touch with [Support](https://support.disruptive-technologies.com/hc/en-us/articles/360010678499).

## 501 - Method Not Allowed <a href="#id-501" id="id-501"></a>

Indicates that the wrong HTTP verb has been used for the request.

This error can be returned for example if you send a `GET`request when the endpoint requires a `POST`request. Double check the [REST API reference docs](https://developer.disruptive-technologies.com/api) to make sure you're using the correct HTTP verb.

## 503 - Service unavailable <a href="#id-503" id="id-503"></a>

See **500 - Internal Server Error**.

## 504 - Gateway timeout <a href="#id-504" id="id-504"></a>

See **500 - Internal Server Error**, but retry with a minimum 10s delay.


# Emulator API

An introduction to our Emulator API, accessible through both DT Studio and REST API.

## Overview

If you don't have access to physical sensors and want to test our services, the Sensor Emulator API gives you the ability to create and experiment with devices at will. An emulated sensor is designed to have all the features of a real sensor, making it a valuable tool for experimentation or debugging.

* Explore DT Studio without the need for real sensors.
* It can be interacted with through the REST API.
* Emit any event type at will by simply clicking a button.
* Generate events in your CI flow to automate integration tests.

You can find a list of all available emulator endpoints at our [REST API Reference](https://developer.disruptive-technologies.com/api/emulator).

## Prerequisites

* **Service Account**\
  Creating, deleting, and interacting with emulated devices require that your User or Service Account has been granted the role of Project Developer or higher.

## Creating an Emulated Device

{% tabs %}
{% tab title="DT Studio" %}
In DT Studio, navigate to the project in which you want to create the emulated sensor. In the left-most menu, locate the **API Integrations -> Emulator** page. Click **Create Device**.

![](/files/-MTZeZj5ds1k6bkI0Mc5)

Under the new device creation page, select a Device Type. A **Device name** and **labels** are optional but can be useful when managing several emulated devices at once.

![](/files/-MUYqNepCnSAsRST3E6R)

Remember to **Save New Emulated Device**.
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://emulator.d21s.com/v2/projects/<PROJECT_ID>/devices`

A request body with the following parameters is required.

```javascript
{
    "type": "temperature"
}
```

A list of all available parameters can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api/emulator).

#### Example Usage

Using cURL with a Service Account for authentication, the following example creates a new emulated temperature sensor with a given name and a "group-number" label.

```bash
curl -X POST "https://emulator.d21s.com/v2/projects/<PROJECT_ID>/devices" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"type": "temperature", "labels": {"name": "my-emulated-sensor", "group-number": "99"}}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a new Emulated Device can be created by calling the following resource method.

[disruptive.Emulator.create\_device()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/emulator.html#create-device)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example creates a new Emulated Temperature Device.

```python
import disruptive as dt

device = dt.Emulator.create_device(
    project_id='<PROJECT_ID>',
    device_type=dt.Device.TEMPERATURE,
)

print(device)
```

{% endtab %}
{% endtabs %}

## Emitting Emulated Events

Events can be emitted at will using an emulated device.

{% tabs %}
{% tab title="DT Studio" %}
In DT Studio, navigate to the project containing your emulated sensors. In the left-most menu, locate the **API Integrations -> Emulator** page. Click on your emulated device.

![](/files/-MT_5vw_yt5v7B5jhAif)

Under the details page of your emulated device, select the **Event Type** and related value you wish to emit. When set, click **Send Event** for immediate effect.

![](/files/-MT_6kjM6fneCSdLJK2_)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://emulator.d21s.com/v2/projects/<PROJECT_ID>/devices/<DEVICE_ID>:publish`

The body of the POST request should be the [event](/concepts/events#variations) to be published. The type of the published event must match the device type of the emulated event. For example, an emulated `temperature` sensor can't send `objectPresent` events. An overview of which events types are available for a given device type can be found [here](/concepts/devices#wireless-sensors).

**Note:** The `updateTime` field is optional for all event types, and defaults to now.

#### Example Usage

As an example, to publish a `temperature` event, include the following body in the POST request.

```json
{
    "temperature": {
        "value": 42
    }
}
```

Using cURL with a Service Account for authentication, the following example emits a temperature event from the specified emulated temperature sensor.

```bash
curl -X POST "https://emulator.d21s.com/v2/projects/<PROJECT_ID>/devices/<DEVICE_ID>:publish" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"temperature": {"value": 42}}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), an emulated event can be published by calling the following resource method.

[disruptive.Emulator.publish\_event()](https://developer.disruptive-technologies.com/api/libraries/python/client/resources/emulator.html#publish-event)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example published a new [motion event](/concepts/events#motion-event) to an emulated [motion sensor](/concepts/devices#motion-sensor).

```python
import disruptive as dt

dt.Emulator.publish_event(
    device_id='<DEVICE_ID>',
    project_id='<PROJECT_ID>',
    data=dt.events.Motion(
        state='MOTION_DETECTED',
    ),
)
```

{% endtab %}
{% endtabs %}


# Examples

A few short examples on how to use our REST API.

Our REST API allows you to manage all of our services programmatically. The following examples showcase a few common implementations often used as part of larger integrations.

* [Pagination](/examples/pagination)
* [Streaming Events](/examples/streaming-events)
* [Touch to Identify](/examples/touch-to-identify)
* [Refreshing Access Token](/examples/refreshing-access-token)


# Pagination

An example of how to use pagination for fetching a lot of data at once from the REST API.

## Overview

If your application needs to fetch more data than a single request can handle, pagination can be used to seamlessly fetch blocks of the data, or pages, until completion. It splits a large request into smaller ones at the cost of a few extra lines of code. While this example shows how one can fetch projects with pagination, it can be adapted for other requests if desired.

## Preliminaries

* **Basic Auth**\
  For simplicity, we here use [Basic Auth](/authentication/basic-auth) for authentication. We recommend replacing this with an [OAuth2 flow](/authentication/oauth2) production-level integrations.
* **Service Account Credentials**\
  You must create and know the credentials of a [Service Account](/service-accounts/introduction-to-service-accounts). Any role will suffice.
* **REST API**\
  This example utilizes our REST API to interact with our cloud. See the [REST API Reference](https://developer.disruptive-technologies.com/api) for a full list of available endpoints.

## Example Code

The following points summarize the provided example code.

* Projects are fetched in groups of 100 at a time.
* When all projects have been fetched, print the total number of projects.
* Iterate through and print the display name for each project.

### Environment Setup

If you wish to run the code locally, make sure you have a working runtime environment.

{% tabs %}
{% tab title="Python 3.11" %}
The following packages are required to run the example code.

```
pip install requests==2.31.0
```

{% endtab %}

{% tab title="Node.js 20" %}
The following modules are required by the example code and must be installed.

```bash
npm install axios@1.4.0
```

{% endtab %}

{% tab title="Go 1.20" %}
No additional packages are required.
{% endtab %}
{% endtabs %}

Add the following environment variables as they will be used to authenticate the API.

```bash
export DT_SERVICE_ACCOUNT_KEY_ID=<YOUR_SERVICE_ACCOUNT_KEY_ID>
export DT_SERVICE_ACCOUNT_SECRET=<YOUR_SERVICE_ACCOUNT_SECRET>
export DT_SERVICE_ACCOUNT_EMAIL=<YOUR_SERVICE_ACCOUNT_EMAIL>
```

### Source

The following code snippet implements pagination in a few languages.

{% tabs %}
{% tab title="Python 3.11" %}

```python
import os
import requests

# Authentication credentials loaded from environment.
SERVICE_ACCOUNT_KEY_ID = os.getenv('DT_SERVICE_ACCOUNT_KEY_ID', '')
SERVICE_ACCOUNT_SECRET = os.getenv('DT_SERVICE_ACCOUNT_SECRET', '')

# Shortform Disruptive REST API base url.
BASE_URL = 'https://api.d21s.com/v2'


def paginated_get(url: str, result_field: str, key_id: str, secret: str):
    results = []

    # Create a parameters dictionary that contains an empty page token.
    params: dict = {'pageToken': ''}

    # Loop until all pages have been fetched.
    print('Paging...')
    while True:
        # Send GET request for projects.
        page = requests.get(url, params=params, auth=(key_id, secret)).json()
        if 'error' in page:
            raise Exception(page)
        elif result_field not in page:
            raise KeyError(f'Field "{result_field}" not in response.')

        # Concatenate the request response to output.
        results += page[result_field]
        print(f'- Got {len(page[result_field])} results in page.')

        # Update parameters dictionary with next page token.
        if len(page['nextPageToken']) > 0:
            params['pageToken'] = page['nextPageToken']
        else:
            break

    return results


if __name__ == '__main__':
    # Make paginated requests for all available projects.
    projects = paginated_get(
        url=BASE_URL + '/projects',
        result_field='projects',
        key_id=SERVICE_ACCOUNT_KEY_ID,
        secret=SERVICE_ACCOUNT_SECRET,
    )

    # Print display name of all fetched projects.
    for i, project in enumerate(projects):
        print('{}. {}'.format(i, project['displayName']))
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
const axios = require('axios').default

// Environment variables for authentication and target.
const serviceAccountKeyId  = process.env.DT_SERVICE_ACCOUNT_KEY_ID
const serviceAccountSecret = process.env.DT_SERVICE_ACCOUNT_SECRET

// Shortform Disruptive REST API base url.
const baseUrl = 'https://api.d21s.com/v2'

async function paginatedGet(url, resultField, keyId, secret) {
    let results = []

    // Create a parameters dictionary that contains an empty page token.
    let params = {'pageToken': ''}

    // Loop until all pages have been fetched.
    console.log('Paging...')
    while (true) {
        // Send GET request for projects.
        let page = await axios({
            method: 'GET',
            url: url,
            auth: {
                username: keyId,
                password: secret,
            },
            params: params,
        }).catch(function (error) {
            if (error.response) {
                throw new Error(
                    error.response.data.code + ' - '
                    + error.response.data.error
                )
            }
        })

        if (!(resultField in page.data)) {
            throw new Error('Field "' + resultField + '" not in response.')
        }

        // Concatenate response contents to output list.
        results.push(...page.data[resultField])
        console.log(`- ${page.data[resultField].length} projects in page.`)

        // Update parameters with next page token.
        if (page.data.nextPageToken.length > 0) {
            params.pageToken = page.data.nextPageToken
        } else {
            break
        }
    }

    return results
}

async function main () {
    // Make paginated requests for all available projects.
    let projects = []
    projects = await paginatedGet(
        baseUrl + '/projects',
        'projects',
        serviceAccountKeyId,
        serviceAccountSecret,
    )

    // Print display name of all fetched projects.
    for (let i = 0; i < projects.length; i++) {
        console.log(`${i}. ${projects[i]['displayName']}`)
    }
}

main().catch((err) => {console.log(err)});

```

{% endtab %}

{% tab title="Go 1.20" %}

```go
package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"
	"time"
)

const (
	baseURL = "https://api.d21s.com/v2"
)

// Read the environment variables used for authentication
var keyID = os.Getenv("DT_SERVICE_ACCOUNT_KEY_ID")
var secret = os.Getenv("DT_SERVICE_ACCOUNT_SECRET")

// Struct that represents a project fetched from the API
type Project struct {
	Name                    string `json:"name"`
	DisplayName             string `json:"displayName"`
	Inventory               bool   `json:"inventory"`
	Organization            string `json:"organization"`
	OrganizationDisplayName string `json:"organizationDisplayName"`
	Sensorcount             int    `json:"sensorCount"`
	CloudConnectorCount     int    `json:"cloudConnectorCount"`
}

// Sends a GET request to the Disruptive REST API and returns the response body.
func getRequest(endpointURL string, queryParams map[string]string) ([]byte, error) {
	// Create the HTTP GET request
	req, err := http.NewRequest("GET", endpointURL, nil)
	if err != nil {
		return nil, err
	}

	// Set the query parameters for the request
	q := req.URL.Query()
	for key, value := range queryParams {
		q.Set(key, value)
	}
	req.URL.RawQuery = q.Encode()

	// Set the Authorization header to use HTTP Basic Auth
	req.SetBasicAuth(keyID, secret)

	// Send the HTTP request with a 3 second timout in case we
	// are unable to reach the server.
	client := &http.Client{Timeout: time.Second * 3}
	response, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer response.Body.Close()

	return ioutil.ReadAll(response.Body)
}

// getAllProjects retrieves all projects that are accessible to the
// Service Account, one page at a time.
func paginatedGetProjects() ([]Project, error) {
	// Initialize empty output slice.
	projects := []Project{}

	// Define the structure of each page.
	type ProjectsPage struct {
		Projects      []Project `json:"projects"`
		NextPageToken string    `json:"nextPageToken"`
	}

	// Start with an empty page token to request the first page.
	// We'll update this variable to point to the subsequent pages.
	var pageToken = ""

	// Loop until all pages have been fetched
	for {
		// Define the query parameters for the request.
		// The "page_token" query parameter specifies the page
		// we want to get. The "page_size" parameter is included
		// just for the sake of completion. The default page size is 100.
		params := map[string]string{
			"page_token": pageToken,
			"page_size":  "100",
		}

		// Send GET request to the endpoint and get bytes back
		pageBytes, err := getRequest(baseURL+"/projects", params)
		if err != nil {
			return nil, err
		}

		// Decode pageBytes into the expected page structure
		page := ProjectsPage{}
		if err := json.Unmarshal(pageBytes, &page); err != nil {
			return nil, err
		}

		// Append the retrieved page of projects to our output slice
		projects = append(projects, page.Projects...)
		fmt.Printf("Got a page of %d projects\n", len(page.Projects))

		// Update the page token to point to the next page
		pageToken = page.NextPageToken

		// If we got an empty "next_page_token" in the response,
		// it means there are no more pages to fetch.
		if len(pageToken) == 0 {
			break
		}
	}

	return projects, nil
}

func main() {
	// Make paginated requests for all available projects.
	projects, err := paginatedGetProjects()
	if err != nil {
		log.Fatal(err)
	}

	// Print display name of all fetched projects.
	for i, project := range projects {
		fmt.Printf("%d. %s\n", i, project.DisplayName)
	}
}
```

{% endtab %}
{% endtabs %}

### Expected Output

The number of projects resulting from each page will be displayed in the stdout.

```
Paging...
- Got 100 projects in page.
- Got 100 projects in page.
- Got 100 projects in page.
- Got 68 projects in page.

0. my-project-A
1. my-project-B
2. my-project-C
3. ...
```


# Streaming Events

An example of how to use the REST API to stream sensor events in real-time.

## Overview

When plotting life graphs or triggering certain alarms, a continuous stream of data is often preferred over periodic polling. In this example, we will see how the REST API can be used to set up a stream with some best practices and a simple retry policy.

## Preliminaries

* **Data Connectors**\
  If you want to forward your data in a server-to-server integration, consider using [Data Connectors](/data-connectors/introduction-to-data-connector) for a simpler and more reliable service with an added [at-least-once guarantee](/data-connectors/introduction-to-data-connector#at-least-once-guarantee).
* **Basic Auth**\
  For simplicity, we here use Basic Auth for authentication. We recommend replacing this with an [OAuth2 flow](/authentication/oauth2) for integrations more complex than local experimentation.
* **Service Account Credentials**\
  You must create and know the credentials of a Service Account. Any role will suffice.
* **REST API**\
  This example utilizes our REST API to interact with our cloud. See the [REST API Reference](https://developer.disruptive-technologies.com/api) for a full list of available endpoints.

### Response Format

Our REST API `:stream` endpoints support two types of response formats, `text/event-stream` and `application/json`. They are set using the `Accept` header, and defaults to `application/json`.

```python
headers = { "Accept": "application/json" }  # default
headers = { "Accept": "text/event-stream" } # alternative
```

The different headers are used in different cases.

* `application/json`: Returns a JSON object for each event, separated by line-breaks. Easy to parse in any high-level language and used in the code sample below.
* `text/event-stream`: A [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) specific format. Used by any Server-Sent Events libraries, such as `EventSource`.

## Stream Best Practices

The following practices should be considered when implementing a stream. They are all used in the following example.

* **Implement a Retry Policy**\
  The connection to a stream can be lost at any moment due to several factors that break the connection and should be handled by implementing a retry policy. The stream will always disconnect after one hour, as that's how long the access token lasts. If you have not received a single event for the full hour, the stream will disconnect with a 408 HTTP status code.
* **Detect Stream Disconnects Early**\
  An optional `ping_interval` query parameter may be used to make sure the client can still receive messages from the server. If no ping messages have been received in the specified interval, the client should reconnect to the stream.
* **Filter Events**\
  By default, all event types are included in the stream. Use query parameters to reduce the number of events that need to be processed.
* **Authorization Header**\
  Use the `Authorization` header when possible. Some libraries—like the built-in `EventSource` class in browsers—does not allow setting headers. In this case, the `token` query parameter can be used instead.

## Example Code

The following points summarize the provided example code.

* Sends a GET request to the REST API to initialize an event stream.
* Keep the TCP connection open while receiving events.
* If the connection is lost or it's been too long between pings, retry N times before giving up.

### Environment Setup

If you wish to run the code locally, make sure you have a working runtime environment.

{% tabs %}
{% tab title="Python 3.11" %}
The following packages are required by the example code.

```
pip install requests==2.31.0
```

{% endtab %}

{% tab title="Python API" %}
The latest version of our [Python API](https://developer.disruptive-technologies.com/api/libraries/python/) can be installed through pip.

```
pip install --upgrade disruptive
```

{% endtab %}

{% tab title="Node.js 20" %}
The following modules are required by the example code and must be installed.

```bash
npm install eventsource@2.0.2
```

{% endtab %}

{% tab title="Go 1.20" %}
The Go example uses only the standard library, so no additional modules are required.&#x20;
{% endtab %}
{% endtabs %}

Add environment variables for authentication details.

```bash
export DT_SERVICE_ACCOUNT_KEY_ID=<YOUR_SERVICE_ACCOUNT_KEY_ID>
export DT_SERVICE_ACCOUNT_SECRET=<YOUR_SERVICE_ACCOUNT_SECRET>
export DT_SERVICE_ACCOUNT_EMAIL=<YOUR_SERVICE_ACCOUNT_EMAIL>
export DT_PROJECT_ID=<YOUR_PROJECT_ID>
```

### Source

The following code snippet implements streaming in a few languages.

{% tabs %}
{% tab title="Python 3.11" %}

```python
import os
import time
import json

import requests

# Service Account credentials.
SERVICE_ACCOUNT_KEY_ID = os.environ.get('DT_SERVICE_ACCOUNT_KEY_ID')
SERVICE_ACCOUNT_SECRET = os.environ.get('DT_SERVICE_ACCOUNT_SECRET')

# Construct API URL.
PROJECT_ID = os.environ.get('DT_PROJECT_ID')
BASE_URL = 'https://api.d21s.com/v2'
DEVICES_STREAM_URL = '{}/projects/{}/devices:stream'.format(
    BASE_URL,
    PROJECT_ID
)

# A few constants to control stream behavior.
MAX_CONNECTION_RETRIES = 5
PING_INTERVAL = 10
PING_JITTER = 2


if __name__ == '__main__':
    # Set up a simple catch-all retry policy.
    nth_retry = 0
    while nth_retry <= MAX_CONNECTION_RETRIES:
        try:
            print('Streaming... Press CTRL+C to terminate.')
            # Set up a stream connection.
            # Connection will timeout and reconnect if no single event
            # is received in an interval of PING_INTERVAL + PING_JITTER.
            stream = requests.get(
                url=DEVICES_STREAM_URL,
                auth=(SERVICE_ACCOUNT_KEY_ID, SERVICE_ACCOUNT_SECRET),
                stream=True,
                timeout=PING_INTERVAL + PING_JITTER,
                params={
                    'event_types': [],
                    'ping_interval': '{}s'.format(PING_INTERVAL),
                },
            )

            # Iterate through the events as they come in (one per line).
            for line in stream.iter_lines():
                # Decode the response payload and break on error.
                payload = json.loads(line.decode('ascii'))
                if 'result' not in payload:
                    raise Exception(payload)
                event = payload['result']['event']

                # Skip ping events.
                if event['eventType'] == 'ping':
                    continue

                # Print events as they arrive.
                print(f'Got {event["eventType"]} event.')

                # Reset retry counter.
                nth_retry = 0

        except KeyboardInterrupt:
            break

        except requests.exceptions.ConnectionError:
            print('Connection lost. Reconnecting...')
            nth_retry += 1

        except Exception as e:
            print(e)

            # Print the error and try again up to MAX_CONNECTION_RETRIES.
            if nth_retry < MAX_CONNECTION_RETRIES:
                print('Something happened. Retry #{}'.format(nth_retry+1))

                # Exponential backoff in sleep time.
                time.sleep(2**nth_retry)
                nth_retry += 1
            else:
                break
```

{% endtab %}

{% tab title="Python API" %}

```python
import os

import disruptive as dt

PROJECT_ID = os.environ.get('DT_PROJECT_ID', '')


if __name__ == '__main__':
    # Enable logging to see what's happening under the hood.
    dt.log_level = 'debug'

    # Initialize a stream generator for all temperature events in a project.
    for e in dt.Stream.event_stream(PROJECT_ID, event_types=[]):
        # Print event information.
        print(f'Got {e.event_type} event.')
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
const EventSource = require("eventsource")

// Service Account credentials
const serviceAccountKeyID  = process.env.DT_SERVICE_ACCOUNT_KEY_ID
const serviceAccountSecret = process.env.DT_SERVICE_ACCOUNT_SECRET

// Construct API URL
const projectID = process.env.DT_PROJECT_ID
const apiBase = 'https://api.d21s.com/v2/'
const devicesStreamUrl = apiBase + `projects/${projectID}/devices:stream`

// Constants
const maxConnectionRetries = 5  // Max retries without any received messages
const pingInterval         = 10 // Expected interval between pings in seconds
const pingJitter           = 2  // Expected ping jitter in seconds

async function main() {
    let retryCount = 0
    let stream
    setupStream()

    // Sets up a timer that will restart the stream if there has passed too 
    // much time between ping events. This timer is reset every time we 
    // receive a ping.
    const pingTimer = setTimeout(() => {
        console.log("Too long between pings. Reconnecting...")
        clearTimeout(pingTimer)
        setTimeout(() => { // Wait a second before reconnecting
            setupStream()
        }, 1000)
    }, (pingInterval + pingJitter) * 1000)

    async function setupStream() {
        // If we've retried too many times without getting any messages, exit
        if (retryCount >= maxConnectionRetries) {
            console.log("Retried too many times. Exiting")
            process.exit(1)
        }
        retryCount += 1
        
        // Close the existing stream if we have one
        if (stream) {
            stream.close()
        }

        console.log('Streaming... Press CTRL+C to exit.')

        // Add query parameters to the URL
        let url = devicesStreamUrl
        url += `?ping_interval=${pingInterval}s` // Specifies ping interval

        // Prepare the "Authorization" header with basic auth.
        // NOTE: This should be implemented using OAuth2 in a production environment.
        const basicAuthStr = `${serviceAccountKeyID}:${serviceAccountSecret}`
        const headers = {
            Authorization: "Basic " + Buffer.from(basicAuthStr).toString("base64")
        }
        
        // Set up a new stream with callback functions for messages and errors
        // Using headers only works external "eventsource" package. In a browser
        // environment, either use polyfill or the "token" query parameter with
        // an access token from OAuth2.
        stream = new EventSource(url, { headers })
        stream.onmessage = handleStreamMessage
        stream.onerror = handleStreamError
    }
    
    function handleStreamError(err) {
        console.error("Got error from stream:")
        console.error(err)
        console.log("Reconnecting...")
        
        clearTimeout(pingTimer)
        setTimeout(() => { // Wait a second before reconnecting
            setupStream()
        }, 1000)
    }
    
    function handleStreamMessage(message) {
        // Parse the payload as JSON
        const data = JSON.parse(message.data)

        // Check if we got an error
        if (data?.error) {
            handleStreamError(data.error)
            return
        }

        // Reset the retry counter now that we've got an event
        retryCount = 0

        // Parse the event object
        const event = data.result.event
        if (event.eventType === "ping") {
            // We got a ping event. Reset the ping timer
            pingTimer.refresh()
        } else {
            // We got a temperature event
            console.log(`Got ${event.eventType} event.`)
        }
    }
}
main().catch((err) => {console.log(err)})
```

{% endtab %}

{% tab title="Go 1.20" %}
Note that this Go example only handles `temperature` events and must be expanded for other types.

```go
package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"os"
	"strings"
	"time"
)

// Events received from the stream will be in this format.
// Either `Result` or `Error` will be set, the other will be `nil`.
type StreamEvent struct {
	Result *struct {
		Event struct {
			EventId    string          `json:"eventId"`
			TargetName string          `json:"targetName"`
			EventType  string          `json:"eventType"`
			Data       json.RawMessage `json:"data"`
			Timestamp  string          `json:"timestamp"`
		} `json:"event"`
	} `json:"result"`
	Error *struct {
		Code    int    `json:"code"`
		Message string `json:"message"`
		Details []struct {
			Help string `json:"help"`
		} `json:"details"`
	} `json:"error"`
}

// A TemperatureEvent will be available in event's `Data`
// when the `EventType` is `temperature`.
type TemperatureEvent struct {
	Temperature struct {
		Celsius    float32 `json:"value"`
		UpdateTime string  `json:"updateTime"`
		Samples    []struct {
			Celsius    float32 `json:"value"`
			SampleTime string  `json:"sampleTime"`
		} `json:"samples"`
	} `json:"temperature"`
}

const (
	pingInterval = time.Second * 10 // How often we want a ping from the server
	pingLeeway   = time.Second * 2  // Allow for pings to arrive a little late
)

var (
	// This timer will time out if we have not received a ping from the
	// server within the expected interval. We'll use this as a sign that
	// we have lost connection to the server, and then reconnect.
	pingTimer = time.NewTimer(pingInterval + pingLeeway)
)

// Assumes that these environment variables are set before running the script.
var (
	serviceAccountKeyID     = os.Getenv("DT_SERVICE_ACCOUNT_KEY_ID")
	serviceAccountKeySecret = os.Getenv("DT_SERVICE_ACCOUNT_SECRET")
	projectID               = os.Getenv("DT_PROJECT_ID")
)

func startStream(eventTypes []string) {
	// Create the request
	url := fmt.Sprintf("https://api.d21s.com/v2/projects/%s/devices:stream", projectID)
	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		log.Fatalf("Unable to create request: %v", err)
	}

	// Set the "Authorization" header with basic auth.
	// NOTE: OAuth2 authentication should be used in production
	req.SetBasicAuth(serviceAccountKeyID, serviceAccountKeySecret)

	// Add query parameters to the request.
	// "pingInterval" tells the server to send us a periodic pings to let us know the connection is still up.
	// "eventTypes" specifies which event types we want to receive. No specified eventTypes implies all types.
	queryParams := req.URL.Query()
	queryParams.Add("pingInterval", fmt.Sprintf("%.0fs", pingInterval.Seconds()))
	for _, eventType := range eventTypes {
		queryParams.Add("eventTypes", eventType)
	}
	req.URL.RawQuery = queryParams.Encode()

	// Connect to the stream. This will hang until we get the first event...
	fmt.Println("Connecting to stream...")
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Printf("Failed to connect to stream: %v\n", err)
		return
	}
	fmt.Printf("Connected to the stream with status code %d\n", resp.StatusCode)

	// Make sure we close the body when we're done with it
	defer resp.Body.Close()

	// Scanner to read each line in the stream
	scanner := bufio.NewScanner(resp.Body)

	// Channel used to receive raw events from the scanner
	eventChan := make(chan []byte)

	// Channel used to know when the stream has disconnected
	doneChan := make(chan struct{})

	// Scan for events.
	// Events are received as JSON blobs separated by "\n". scanner.Scan() will return
	// true when it has received a new "\n", and return false when the stream disconnects.
	// The event payload is available in scanner.Bytes().
	go func() {
		for scanner.Scan() {
			eventChan <- scanner.Bytes()
		}
		doneChan <- struct{}{}
	}()

	// Wait for either the pingTimer to go off, raw events to come in, or body to be closed.
	for {
		select {
		case <-pingTimer.C:
			fmt.Println("Missed ping. Reconnecting...")
			return
		case rawEvent := <-eventChan:
			handleRawEvent(rawEvent)
		case <-doneChan:
			fmt.Printf("Stream disconnected. Error: %v\n", scanner.Err())
			return
		}
	}
}

func handleRawEvent(rawEvent []byte) {
	// Unmarshal the stream event
	var streamEvent StreamEvent
	if err := json.Unmarshal(rawEvent, &streamEvent); err != nil {
		log.Fatalf("Unable to unmarshal stream payload %s: %v", string(rawEvent), err)
	}

	if streamEvent.Error != nil {
		fmt.Printf("Got error message: %s\n", streamEvent.Error.Message)
		return
	}

	event := streamEvent.Result.Event

	// Check first if we got a ping event
	if event.EventType == "ping" {
		// We got a ping, so we know the stream is still up
		pingTimer.Reset(pingInterval + pingLeeway)
		return
	}

	// The `targetName` field has the format: "projects/<PROJECT_ID>/devices/<DEVICE_ID>"
	deviceID := strings.Split(event.TargetName, "/")[3]

	// Figure out which event type we got
	switch event.EventType {
	case "temperature":
		// Unmarshal temp data
		var tempEvent TemperatureEvent
		if err := json.Unmarshal(event.Data, &tempEvent); err != nil {
			log.Fatalf("Unable to unmarshal temperature event %s: %v", string(event.Data), err)
		}
		handleTempEvent(tempEvent, deviceID)
	default:
		// Not yet handling this event type
		fmt.Printf("Got %s event -> %s\n", event.EventType, string(event.Data))
	}
}

func handleTempEvent(tempEvent TemperatureEvent, deviceID string) {
	timestamp, err := time.Parse(time.RFC3339, tempEvent.Temperature.UpdateTime)
	if err != nil {
		log.Fatalf("Unable to parse temperature timestamp %s: %v", tempEvent.Temperature.UpdateTime, err)
	}

	fmt.Printf("%.2f°C from %s at %s\n",
		tempEvent.Temperature.Celsius,
		deviceID,
		timestamp,
	)

	// Do more processing here...
}

func main() {
	// Always keep the stream connected. Wait 1 sec before reconnecting
	for {
		startStream([]string{"temperature"})
		time.Sleep(time.Second)
		fmt.Println("Reconnecting...")
	}
}
```

{% endtab %}
{% endtabs %}

### Expected Output

For each new event in the stream, a line will be printed to stdout.

```bash
Streaming... Press CTRL+C to terminate.
Got touch event.
Got networkStatus event.
...
```


# Touch to Identify

An example of how to use the REST API with a simple script that identifies devices by touch.

## Overview

A feature in DT Studio popular with our customers is the ability to locate a sensor by simply touching it. Like all functions in DT Studio, the implementation is based on our REST API. We will in this example show how simple it is to recreate using the stream endpoint.

## Preliminaries

* **Basic Auth**\
  For simplicity, we here use Basic Auth for authentication. We recommend replacing this with an [OAuth2 flow](/authentication/oauth2) for integrations more complex than local experimentation.
* **Service Account Credentials**\
  You must create and know the credentials of a Service Account. Any role will suffice.
* **Streaming Best Practices**\
  While this example is based on our other example for [Streaming Events](/examples/streaming-events), it does not implement the same retry policy or other best practices as it is not the focus here. You are, however, free to combine the two examples for a more robust touch-event-listening loop.

## Example Code

The following points summarize the provided example code.

* Sends a GET request to the REST API to initialize an event stream.
* Keep the TCP connection open while receiving events.
* When receiving a Touch Event, fetch and print the source device information.
* Break the stream.

### Environment Setup

If you wish to run the code locally, make sure you have a working runtime environment.

{% tabs %}
{% tab title="Python 3.11" %}
The following packages are required by the example code and must be installed.

```
pip install requests==2.31.0
```

{% endtab %}

{% tab title="Python API" %}
The latest version of our [Python API](https://developer.disruptive-technologies.com/api/libraries/python/) can be installed through pip.

```
pip install --upgrade disruptive
```

{% endtab %}

{% tab title="Node.js 20" %}
The following modules are required by the example code and must be installed.

```bash
npm install eventsource@2.0.2
npm install jsonwebtoken@9.0.1
npm install axios@1.4.0
```

{% endtab %}
{% endtabs %}

Add the following environment variables as they will be used to authenticate the API.

{% tabs %}
{% tab title="Bash" %}

```bash
export DT_SERVICE_ACCOUNT_KEY_ID=<YOUR_SERVICE_ACCOUNT_KEY_ID>
export DT_SERVICE_ACCOUNT_SECRET=<YOUR_SERVICE_ACCOUNT_SECRET>
export DT_SERVICE_ACCOUNT_EMAIL=<YOUR_SERVICE_ACCOUNT_EMAIL>
export DT_PROJECT_ID=<YOUR_PROJECT_ID>
```

{% endtab %}
{% endtabs %}

### Source

The following code snippet implements the touch-to-identify listening loop.

{% tabs %}
{% tab title="Python 3.11" %}

```python
import os
import json

import requests

# Service Account credentials.
SERVICE_ACCOUNT_KEY_ID = os.getenv('DT_SERVICE_ACCOUNT_KEY_ID')
SERVICE_ACCOUNT_SECRET = os.getenv('DT_SERVICE_ACCOUNT_SECRET')

# Construct API URL.
PROJECT_ID = os.getenv('DT_PROJECT_ID')
API_BASE = 'https://api.d21s.com/v2/'
DEVICES_STREAM_URL = '{}projects/{}/devices:stream'.format(
    API_BASE,
    PROJECT_ID
)

if __name__ == '__main__':
    # Set up a stream connection.
    print('Waiting for touch event...')
    stream = requests.get(
        url=DEVICES_STREAM_URL,
        auth=(SERVICE_ACCOUNT_KEY_ID, SERVICE_ACCOUNT_SECRET),
        stream=True,
        params={
            'event_types': ['touch'],
        },
    )

    # Iterate through the events as they come in (one event per line).
    for line in stream.iter_lines():
        # Decode the response payload and isolate event dictionary.
        payload = json.loads(line.decode('ascii'))

        # Halt at missing key.
        if 'result' not in payload.keys():
            print(payload)
            break

        # Fetch touched device blob for more detailed information.
        event = payload['result']['event']
        device = requests.get(
            url=API_BASE + event['targetName'],
            auth=(SERVICE_ACCOUNT_KEY_ID, SERVICE_ACCOUNT_SECRET),
        ).json()

        # Print some device information.
        print('\nTouch event received:')
        print(json.dumps(device, indent=4))

        # Stop stream as we've found a device.
        break
```

{% endtab %}

{% tab title="Python API" %}

```python
import os

import disruptive as dt

# Environment variables for target project.
PROJECT_ID = os.getenv('DT_PROJECT_ID', '')


if __name__ == '__main__':
    # Enable logging to see what's happening under the hood.
    dt.log_level = 'debug'

    # Set up a stream for all touch events in a project.
    for e in dt.Stream.event_stream(PROJECT_ID, event_types=['touch']):
        # When a touch event is received, fetch the source device.
        touched_device = dt.Device.get_device(e.device_id)

        # Print information about the touched device.
        print(touched_device)

        # Stop the stream as we've found a device.
        break
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
const EventSource = require("eventsource") // npm install eventsource@1.0.7
const jwt = require('jsonwebtoken')        // npm install jsonwebtoken@8.5.1
const axios = require('axios').default     // npm install axios@0.21.1

// Service Account credentials
const serviceAccountKeyID  = process.env.DT_SERVICE_ACCOUNT_KEY_ID
const serviceAccountSecret = process.env.DT_SERVICE_ACCOUNT_SECRET
const serviceAccountEmail  = process.env.DT_SERVICE_ACCOUNT_EMAIL

// Construct API URL
const projectID = process.env.DT_PROJECT_ID
const apiBase = 'https://api.d21s.com/v2/'
const devicesStreamUrl = apiBase + `projects/${projectID}/devices:stream`

async function main() {

    let retryCount = 0
    let stream
    setupStream()

    async function setupStream() {
        // Since EventSource does not support setting HTTP headers, we need to use
        // OAuth for authentication. The received access token will be set as a
        // `token` query parameter when setting up the stream in the next step.
        const accessToken = await getAccessToken(
            serviceAccountKeyID, 
            serviceAccountEmail, 
            serviceAccountSecret,
        )
        
        // Add query parameters to the URL
        let url = devicesStreamUrl
        url += `?eventTypes=touch`     // Filters out all events except touch
        url += `&token=${accessToken}` // The access token for authentication

        // Set up a new stream with callback functions for messages
        console.log('Waiting for touch event...')
        stream = new EventSource(url)
        stream.onmessage = handleStreamMessage
    }
    
    async function handleStreamMessage(message) {
        // Parse the payload as JSON
        const data = JSON.parse(message.data)
        if (!data || !data.result) {
            return
        }

        // Parse the event object
        const event = data.result.event

        // Close stream as we've found a device
        stream.close()

        // Send GET request to fetch device information.
        const response = await axios({
            method: 'GET',
            url: apiBase + event.targetName,
            auth: {
                username: process.env.DT_SERVICE_ACCOUNT_KEY_ID,
                password: process.env.DT_SERVICE_ACCOUNT_SECRET,
            }
        })

        // Print response contents.
        console.log('\nTouch event received:')
        console.log(JSON.stringify(response.data, null, 2))
    }
}
main().catch((err) => {console.log(err)})

// Fetches an access token. See the following guide for documentation:
// https://developer.d12s.com/docs/authentication/oauth2
async function getAccessToken(keyID, email, secret) {
    // Construct the JWT header
    let jwtHeaders = {
        'alg': 'HS256',
        'kid': keyID,
    }
    
    // Construct the JWT payload
    let jwtPayload = {
        'iat': Math.floor(Date.now() / 1000),        // current unixtime
        'exp': Math.floor(Date.now() / 1000) + 3600, // expiration unixtime
        'aud': 'https://identity.disruptive-technologies.com/oauth2/token',
        'iss': email,
    }
    
    // Sign and encode JWT with the secret
    const jwtEncoded = jwt.sign(
        jwtPayload,
        secret,
        {
            header: jwtHeaders,
            algorithm: 'HS256',
        },
    )

    // Prepare POST request data
    const requestObject = {
        'assertion': jwtEncoded,
        'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer',
    }

    // Converts the requestObject to a Form URL-Encoded string
    const requestData = Object.keys(requestObject).map(function(key) {
        return encodeURIComponent(key)+'='+encodeURIComponent(requestObject[key])
    }).join('&')
    
    // Exchange JWT for access token
    const accessTokenResponse = await axios({
        method: 'POST',
        url: 'https://identity.disruptive-technologies.com/oauth2/token',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
        data: requestData,
    }).catch(function (error) {
        if (error.response) {
            console.log(error.response.data)
        }
        throw error
    })

    // Return the access token in the request
    return accessTokenResponse.data.access_token
}
```

{% endtab %}
{% endtabs %}

### Expected Output

Once a touch event has been received, the device information is printed and stream terminated.

```
Waiting for touch event...

Touch event received:
{
    "name": "projects/c0md3mm0c7bet3vico8g/devices/emuc0uc989qdqebrvv29so0",
    "type": "touch",
    "labels": {
        "name": "my-favorite-sensor",
        "virtual-sensor": ""
    },
    "reported": {
        "networkStatus": {
            "signalStrength": 99,
            "rssi": 0,
            "updateTime": "2021-03-03T17:21:53.080314Z",
            "cloudConnectors": [
                {
                    "id": "emulated-ccon",
                    "signalStrength": 99,
                    "rssi": 0
                }
            ],
            "transmissionMode": "LOW_POWER_STANDARD_MODE"
        },
        "batteryStatus": null,
        "touch": {
            "updateTime": "2021-03-03T17:22:51.059514Z"
        }
    }
}

Terminating stream.
```


# Refreshing Access Token

An example of how the authentication access token can be refreshed automatically.

## Overview

When running integrations continuously, authentication towards the API must be refreshed periodically. By default, this must be done every hour and should happen automatically without interfering with the integration. In this example,  a simple access token refresh routine that keeps the API authenticated has been implemented.

## Preliminaries

* **Service Account Credentials**\
  You must create and know the credentials of a [Service Account](/service-accounts/creating-a-service-account). Any role will suffice.
* **OAuth2**\
  This example heavily builds on the [OAuth2 Authentication Guide](/authentication/oauth2). It is recommended that you understand this routine before attempting to expand upon it.

## Example Code

The following points summarize the provided example code.

* A simulated REST API call is triggered every 5 seconds.
* An instance of the Auth class provides the access token as required.
* If the access token is within 1 minute of expiration, the token is first refreshed.

### Environment Setup

If you wish to run the code locally, make sure you have a working runtime environment.

{% tabs %}
{% tab title="Python 3.11" %}
The following packages are required by the example code and must be installed.

```
pip install pyjwt==2.7.0
pip install requests==2.31.0
```

{% endtab %}

{% tab title="Node.js 20" %}
The following packages are required by the example code and must be added.

```
npm install jsonwebtoken@9.0.1
npm install axios@1.4.0
```

{% endtab %}
{% endtabs %}

Add the following credentials to your environment as they will be used to authenticate the API.

```bash
export DT_SERVICE_ACCOUNT_KEY_ID=<YOUR_SERVICE_ACCOUNT_KEY_ID> # [string]
export DT_SERVICE_ACCOUNT_EMAIL=<YOUR_SERVICE_ACCOUNT_EMAIL>   # [string]
export DT_SERVICE_ACCOUNT_SECRET=<YOUR_SERVICE_ACCOUNT_SECRET> # [string]
```

### Source

The following code snippet implements the token refresh routine.

{% tabs %}
{% tab title="Python 3.11" %}

```python
import os
import time
import jwt       # pip install pyjwt==2.1.0
import requests  # pip install requests==2.25.1

# Constants for authentication credentials.
KEY_ID = os.environ.get('DT_SERVICE_ACCOUNT_KEY_ID', '')
SECRET = os.environ.get('DT_SERVICE_ACCOUNT_SECRET', '')
EMAIL = os.environ.get('DT_SERVICE_ACCOUNT_EMAIL', '')


class Auth():
    """
    Handles automatic refresh of access token every
    time get_token() is called with a buffer of 1 minute.

    """

    refresh_buffer = 60  # s
    auth_url = 'https://identity.disruptive-technologies.com/oauth2/token'

    def __init__(self, key_id: str, email: str, secret: str):
        # Set attributes.
        self.key_id = key_id
        self.email = email
        self.secret = secret

        # Initialize some variables.
        self.token = ''
        self.expiration = 0

    def refresh(self):
        # Construct the JWT header.
        jwt_headers = {
            'alg': 'HS256',
            'kid': self.key_id,
        }

        # Construct the JWT payload.
        jwt_payload = {
            'iat': int(time.time()),         # current unixtime
            'exp': int(time.time()) + 3600,  # expiration unixtime
            'aud': self.auth_url,
            'iss': self.email,
        }

        # Sign and encode JWT with the secret.
        encoded_jwt = jwt.encode(
            payload=jwt_payload,
            key=self.secret,
            algorithm='HS256',
            headers=jwt_headers,
        )

        # Prepare HTTP POST request data.
        # note: The requests package applies Form URL-Encoding by default.
        request_data = {
            'assertion': encoded_jwt,
            'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer'
        }

        # Exchange the JWT for an access token.
        access_token_response = requests.post(
            url=self.auth_url,
            headers={'Content-Type': 'application/x-www-form-urlencoded'},
            data=request_data,
        )

        # Halt if response contains an error.
        if access_token_response.status_code != 200:
            print('Status Code: {}'.format(access_token_response.status_code))
            print(access_token_response.json())
            return None

        # Unpack the response dictionary.
        token_json = access_token_response.json()
        self.token = token_json['access_token']
        self.expiration = time.time() + token_json['expires_in']

    def get_token(self):
        # Check if access token needs a refresh. A 1 minute buffer is added.
        if self.expiration - time.time() < self.refresh_buffer:
            # Print expiration message to console.
            print('Refreshing...')

            # Fetch a brand new access token and expiration.
            self.refresh()

        # Print time until expiration.
        print('Token expires in {}s.'.format(
            int(self.expiration - time.time() - self.refresh_buffer),
        ))

        # Return the token to user.
        return self.token


def function_that_calls_rest_api(access_token: str):
    # This would usually do something useful.
    time.sleep(5)


if __name__ == '__main__':
    # Initialize an authentication object.
    auth = Auth(KEY_ID, EMAIL, SECRET)

    # Do some task, here simulated by an infinite loop.
    while True:
        # Simulate some routine that needs authentication for the REST API.
        function_that_calls_rest_api(auth.get_token())
```

{% endtab %}

{% tab title="Node.js 20" %}

```javascript
const jwt = require('jsonwebtoken')
const axios = require('axios').default

// Authentication credentials.
const serviceAccountKeyID  = process.env.DT_SERVICE_ACCOUNT_KEY_ID
const serviceAccountSecret = process.env.DT_SERVICE_ACCOUNT_SECRET
const serviceAccountEmail  = process.env.DT_SERVICE_ACCOUNT_EMAIL

// Token refresh variables and constants.
const refreshBuffer = 60 // seconds
let expiration      = 0  // unixtime
let accessToken     = ''

async function getAccessToken(keyID, secret, email) {
    // Check if access token needs a refresh. A buffer is added.
    if (expiration - Math.floor(Date.now() / 1000) < refreshBuffer) {
        // Print expiration message to console.
        console.log('Refreshing...')

        // Fetch a new token response.
        let res = await refreshAccessToken(keyID, secret, email)

        // Update token and expiration.
        expiration = Math.floor(Date.now() / 1000) + res.expires_in
        accessToken = res.access_token
    }

    // Print time until expiration.
    let sLeft = expiration - Math.floor(Date.now() / 1000) - refreshBuffer
    console.log(`Token expires in ${sLeft}s.`)
}

async function refreshAccessToken(keyID, secret, email) {
    // Construct the JWT header.
    let jwtHeaders = {
        'alg': 'HS256',
        'kid': keyID,
    }
    
    // Construct the JWT payload.
    let jwtPayload = {
        'iat': Math.floor(Date.now() / 1000),        // current unixtime
        'exp': Math.floor(Date.now() / 1000) + 3600, // expiration unixtime
        'aud': 'https://identity.disruptive-technologies.com/oauth2/token',
        'iss': email,
    }
    
    // Sign and encode JWT with the secret.
    const jwtEncoded = jwt.sign(
        jwtPayload,
        secret,
        {
            header: jwtHeaders,
            algorithm: 'HS256',
        },
    )

    // Prepare POST request data.
    const requestObject = {
        'assertion': jwtEncoded,
        'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer',
    }

    // Converts the requestObject to a Form URL-Encoded string.
    const requestData = Object.keys(requestObject).map(function(key) {
        return encodeURIComponent(key) + '=' + encodeURIComponent(requestObject[key])
    }).join('&')
    
    // Exchange JWT for access token.
    const accessTokenResponse = await axios({
        method: 'POST',
        url: 'https://identity.disruptive-technologies.com/oauth2/token',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
        data: requestData,
    }).catch(function (error) {
        // Prints the error response (if any), an re-throws the error.
        if (error.response) {
            console.log(error.response.data)
        }
        throw error
    })

    // Return the response data.
    return accessTokenResponse.data
}

async function functionThatCallsAPI(accessToken) {
    // Use access token for calling the Rest API here.
    // We'll just sleep for a few seconds.
    await new Promise(r => setTimeout(r, 5000));
}

async function main() {
    // Do some task, here simulated by an infite loop.
    while (true) {
        // Call some routine that needs the access token.
        await functionThatCallsAPI(getAccessToken(
            serviceAccountKeyID,
            serviceAccountSecret,
            serviceAccountEmail,
        ))
    }
}
main()

```

{% endtab %}
{% endtabs %}

### Expected Output

For visualization purposes, an expiration timer is printed each time the token is used.

```
Refreshing...
Token expires in 3538s.
Token expires in 3533s.
Token expires in 3528s.
...
Token expires in 13s.
Token expires in 8s.
Token expires in 3s.
Refreshing...
Token expires in 3538s.
Token expires in 3533s
...
```


# Introduction to Service Accounts

A general overview of what Service Accounts are and why we use them.

## Overview

All programmatic interaction with the Disruptive Technologies API is done via a logged-in Service Account. Once a client has been authenticated, the program interacting with the API will be able to perform operations that are within the role access level granted to the Service Account.

## Service and User Accounts

There are two types of accounts available when interacting with DT Cloud, user and Service Account. A user account represents a person and is used for logging in to DT Studio. On the other hand, a Service Account represents a program and is used for machine-to-machine interaction.&#x20;

![](/files/-MTehPLyB4lfniqHgYfd)

Both types of accounts can be a member of one or more project and organization with a specific role. The role they have determines which actions they have permission to perform within the project or organization.

While a user account use an email and password to log in to the browser, a Service Account authenticates itself using a key-secret pair that can be rotated when necessary.


# Creating a Service Account

A quick guide on how to create a Service Account using DT Studio or our REST API.

## Overview

We will here create a new a new Service Account using either DT Studio or our REST API. Once created, the Service Account is granted membership in the project and provided it a role. Then, a new Key Pair is generated as credentials for interacting with the REST API.

## Prerequisites

* **Service Account**\
  Creating, deleting, and interacting with Service Accounts require that your User or existing [Service Account](/service-accounts/introduction-to-service-accounts) has been granted the [role](/service-accounts/managing-access-rights#roles-and-permissions) of Project Administrator or higher.

## New Service Account

The project our new Service Account is created in becomes the owning project. However, this does not provide rights in said project, which must be explicitly granted after creation.

{% tabs %}
{% tab title="DT Studio" %}
In [DT Studio](https://studio.disruptive-technologies.com/), navigate to your Project. In the left menu, locate **Service Accounts** and press **Create new Service Account**. Give it a name and click **Add**.

![](/files/-MT5ksj0L9qk_aZRM3sP)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/serviceaccounts`

A request body is not required and will result in a default configuration. A list of all available parameters can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api).

#### Example Usage

Using cURL with a Service Account for authentication, the following example creates a new Service Account with a given name and [Basic Auth](/authentication) enabled.

```bash
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/serviceaccounts" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"displayName": "my-new-service-account", "enableBasicAuth": true}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a new Service Account can be created by calling the following resource method.

* [disruptive.ServiceAccount.create\_service\_account()](https://developer.disruptive-technologies.com/api/libraries/python/resources/service_account.html#create-service-account)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example creates a new Service Account with a given name and [Basic Auth](/authentication) enabled.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Create a new Service Account in the specified project.
new_sa = dt.ServiceAccount.create_service_account(
    project_id='<PROJECT_ID>',
    display_name='my-new-service-account',
    basic_auth_enabled=True,
)

# Print the newly created Service Account.
print(new_sa)
```

{% endtab %}
{% endtabs %}

## New Project Membership

Your new Service Account is now active but does not have permissions in any projects. We will now give it membership in the project, including a role and other configurations.

{% tabs %}
{% tab title="DT Studio" %}
Click on your new Service Account. This will take you to the configuration page, where the following details are presented. Edit as desired.

* **Service Account Email**\
  An automatically generated email is used for both authentication and access rights management in other projects and organizations. This can not be edited.
* **Role in the current project**\
  Controls which permissions are granted in the current project. You can find a list of all permissions per role on our [Managing Access Rights](/service-accounts/managing-access-rights#roles-and-permissions) page.
* **Enable Basic Auth**\
  The simplest method for authenticating the REST API. While we recommend using an [OAuth2 flow](/authentication/oauth2), Basic Auth can be handy for quick prototyping and single calls.

![](/files/-MT5lFGMql8O3p8Su8km)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/members`

A request body with the following parameters is required.

```javascript
{
    "roles": [
        "roles/<ROLE>"
        // See [Managing Access Rights] for a list of all roles.
    ],
    "email": "<SERVICE_ACCOUNT_EMAIL>"
}
```

A list of all available parameters can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api).

#### Example Usage

Using cURL with a Service Account for authentication, the following example grants the role of Project Developer to a Service Account specified by its email.&#x20;

```bash
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/members" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"roles": ["roles/project.developer"], "email": "<SERVICE_ACCOUNT_EMAIL>"}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Service Account can be granted a project role by calling the following resource method.

* [disruptive.Project.add\_member()](https://developer.disruptive-technologies.com/api/libraries/python/resources/project.html#project-add-member)

The same call exists for an organization under [disruptive.Organization.add\_member()](https://developer.disruptive-technologies.com/api/libraries/python/resources/organization.html#org-add-member).

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example grants the role of Project Developer to a Service Account specified by its email.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Grant Service Account membership in the specified project.
member = dt.Project.add_member(
    project_id='<PROJECT_ID>',
    email='<SERVICE_ACCOUNT_EMAIL>',
    roles=[dt.Role.PROJECT_DEVELOPER],
)

# Print the updated member.
print(member)
```

{% endtab %}
{% endtabs %}

## Generating Keys

The last step is to create a key. Remember that while the generated Key ID will always be listed under your Service Account, **the secret will be shown only once, so make sure to write it down**.

{% tabs %}
{% tab title="DT Studio" %}
On your Service Account configuration page, click **Create New** next to **Active Keys**. The pop-up dialog contains the newly created Key ID and secret.

![](/files/-MT5lWtq2KorGgs0Th7i)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/serviceaccounts/<SERVICE_ACCOUNT>/keys`

No request body should be provided will the request. However, the response contains both the Key ID and secret. Remember to write the secret down.

```javascript
{
  "key": {
    "name": "projects/<PROJECT_ID>/serviceaccounts/<SERVICE_ACCOUNT_ID>/keys/<KEY_ID>",
    "id": "<KEY_ID>",
    "createTime": "2021-02-16T11:09:16.240828Z"
  },
  "secret": "<SECRET>"
}
```

#### Example Usage

Using cURL with a Service Account for authentication, the following example generates a new key for the specified Service Account.

```bash
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/serviceaccounts/<SERVICE_ACCOUNT_ID>/keys" \
    --user "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a new key can be created by calling the following resource method.

* [disruptive.ServiceAccount.create\_key()](https://developer.disruptive-technologies.com/api/libraries/python/resources/service_account.html#create-key)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example generates a new key for the specified Service Account.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Create a new key for the specified Service Account.
key = dt.ServiceAccount.create_key(
    service_account_id='<SERVICE_ACCOUNT_ID>',
    project_id='<PROJECT_ID>'
)

# Print the newly created Service Account key.
print(key)
```

{% endtab %}
{% endtabs %}

## Using Your Service Account

The Service Account creation is now complete, and you may use it as desired.

* [Manage Access Rights in your organization](/service-accounts/managing-access-rights).
* [Explore the REST API using cURL](/explore-our-rest-api/with-curl).
* [Authenticate the REST API with OAuth2](/authentication/oauth2).


# Managing Access Rights

A more technical look at the details regarding Service Account management.

## Hierarchy

Service Accounts are designed to be flexible enough to accommodate most organizational structures. They can be given access to any number of projects or organizations as required, with a specific level of permission in each. The access rights hierarchy can be summarized in a few points.

* A [Service Account](/service-accounts/managing-access-rights#manage-service-accounts) is created under a project which becomes its owner.
* A Service Account can be a [member](/service-accounts/managing-access-rights#members) of other projects and organizations.
* The Service Account membership is assigned a [role](/service-accounts/managing-access-rights#roles-and-permissions) for that project or organization.
* A role provides the member with [permissions](/service-accounts/permissions) within the related project or organization.

![](/files/-MTame0-YNbP1NUdZqV5)

## Manage Service Accounts

When first creating a Service Account within a project, the Service Account will not be a member of any projects or organizations, including the owning project. In DT Studio, the “Role in current Project” option will show as “No access” and can be changed to give the Service Account access to the resources in this project.

### Creating a Service Account

Please refer to our introductory guide on [Creating a Service Account](/service-accounts/creating-a-service-account).

### Deleting a Service Account

{% tabs %}
{% tab title="DT Studio" %}
In your project, navigate to the **API Integrations -> Service Accounts** page. Locate the Service Account you wish to remove, then click the **Remove** button.

![](/files/-MToLGGi5oKD19rmbd3a)
{% endtab %}

{% tab title="REST API" %}
Send a **DELETE** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/serviceaccount/<SERVICEACCOUNT_ID>`

#### Example

Using cURL with a Service Account for authentication, the following example deletes the Service Account specified by its ID.

```bash
curl -X DELETE "https://api.d21s.com/v2/projects/<PROJECT_ID>/serviceaccounts/<DELETING_SERVICE_ACCOUNT_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a Service Account can be deleted by calling the following resource method.

* [disruptive.ServiceAccount.delete\_service\_account()](https://developer.disruptive-technologies.com/api/libraries/python/resources/service_account.html#delete-service-account)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example deletes the Service Account specified by its ID.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Delete the specified Service Account.
dt.ServiceAccount.delete_service_account(
    service_account_id='<SERVICE_ACCOUNT_ID>',
    project_id='<PROJECT_ID>',
)
```

{% endtab %}
{% endtabs %}

## Members

A single Service Account can be a member of several projects and organizations. These members are unique and independent from each other, allowing for different roles to be assigned.

### Manage Project Members

A User or Service Account must have the [role](/service-accounts/managing-access-rights#roles) of Project Admin or higher to manage members.

#### New Project Member

If you do not possess a Service Account with sufficient access rights, new Project members must be added by an existing Project Admin in DT Studio or through the APIs.

{% tabs %}
{% tab title="DT Studio" %}
In your project, navigate to the **Project Settings** page. Here, using the email of the target Service Account, select a role and click **Invite Member**. This can be changed later.&#x20;

![](/files/-MT5ns-P6h4jVYUHhpcS)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/members`

A request body with the following parameters is required.

```javascript
{
    "roles": [
        "roles/<ROLE>"
    ],
    "email": "<SERVICE_ACCOUNT_EMAIL>"
}
```

A list of all available parameters can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api) and a list of all roles and their permissions under the [Roles](/service-accounts/managing-access-rights#roles) subsection on this very page.

#### Example Usage

Using cURL with a Service Account for authentication, the following example grants the specified Service Account the role of Project Developer in the specified project.&#x20;

```bash
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/members" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"roles": ["roles/project.developer"], "email": "<SERVICE_ACCOUNT_EMAIL>"}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a new project member can be added by calling the following resource method.

* [disruptive.Project.add\_member()](https://developer.disruptive-technologies.com/api/libraries/python/resources/project.html#project-add-member)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example grants the specified Service Account the role of Project Developer in a project.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Add a new project developer member to the specified project.
member = dt.Project.add_member(
    project_id='<PROJECT_ID>',
    email='<SERVICE_ACCOUNT_EMAIL>',
    roles=[dt.Role.PROJECT_DEVELOPER],
)

# Print information about the newly created member.
print(member)
```

{% endtab %}
{% endtabs %}

#### Remove Project Member

{% tabs %}
{% tab title="DT Studio" %}
In your project, navigate to the **Project Settings** page. Locate the member you wish to remove, then click the **Remove** button. For reference, see the image under **New Project Member**.
{% endtab %}

{% tab title="REST API" %}
Send a **DELETE** request to:

`https://api.d21s.com/v2/projects/<PROJECT_ID>/members/<MEMBER_ID>`

Here, `<MEMBER_ID>` is either the ID of a Service Account or User.

#### Example Usage

Using cURL with a Service Account for authentication, the following example removes the specified Service Accounts membership in the specified project.

```bash
curl -X DELETE "https://api.d21s.com/v2/projects/<PROJECT_ID>/members/<SERVICE_ACCOUNT_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), a project member can be removed by calling the following resource method.

* [disruptive.Project.remove\_member()](https://developer.disruptive-technologies.com/api/libraries/python/resources/project.html#remove-member)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example removes the specified Service Accounts membership in the specified project.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Remove member from the specified project.
dt.Project.remove_member(
    member_id='<MEMBER_ID>',
    project_id='<PROJECT_ID>',
)
```

{% endtab %}
{% endtabs %}

### Manage Organization Members

A User or Service Account must have the [role](/service-accounts/managing-access-rights#roles) of Organization Admin or higher to manage members.

#### New Organization Member

If you do not possess a Service Account with sufficient access rights, new Organization members must be added by an existing Organization Admin in DT Studio or through the APIs.

{% tabs %}
{% tab title="DT Studio" %}
In any project, navigate to the **Administrators** page. Currently, the only role available for an organization member is `organization.admin`, hence the naming conventions used in DT Studio. Using the email of the Service Account you wish to add, click **Invite Administrator**.

Note that an `organization.admin` will obtain the same permissions as a `project.admin` in every single project under the organization in question.&#x20;

![](/files/-MT5oGjW-JQ4hmcCjQAr)
{% endtab %}

{% tab title="REST API" %}
Send a **POST** request to:

`https://api.d21s.com/v2/organizations/<ORGANIZATION_ID>/members`

A request body with the following parameters is required.

```javascript
{
    "roles": [
        "roles/<ROLES>"
    ],
    "email": "<SERVICE_ACCOUNT_EMAIL>"
}
```

A list of all available parameters can be found in our [REST API Reference](https://developer.disruptive-technologies.com/api). Currently, the only role availbale is `organization.admin`.

#### Example Usage

Using cURL with a Service Account for authentication, the following example grants the specified Service Account the role of Organization Admin in the specified organization.&#x20;

```bash
curl -X POST "https://api.d21s.com/v2/organizations/<ORGANIZATION_ID>/members" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"roles": ["roles/organization.admin"], "email": "<SERVICE_ACCOUNT_EMAIL>"}'
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), an organization member can be added by calling the following resource method.

* [disruptive.Organization.add\_member()](https://developer.disruptive-technologies.com/api/libraries/python/resources/organization.html#org-add-member)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example grants the specified Service Account the role of Organization Admin.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Add a new organization admin member to the organization.
member = dt.Organization.add_member(
    organization_id='<ORGANIZATION_ID>',
    email='<SERVICE_ACCOUNT_EMAIL>',
    roles=[dt.Role.ORGANIZATION_ADMIN],
)

# Print information about the newly created member.
print(member)
```

{% endtab %}
{% endtabs %}

#### Remove Organization Member

{% tabs %}
{% tab title="DT Studio" %}
In any project, navigate to the **Administrators** page. Locate the member you wish to remove, then click the **Remove** button. For reference, see the image under **New Organization Member**.
{% endtab %}

{% tab title="REST API" %}
Send a **DELETE** request to:

`https://api.d21s.com/v2/organizations/<ORGANIZATION_ID>/members/<MEMBER_ID>`

Here, `<MEMBER_ID>` is either the ID of a Service Account or User.

#### Example Usage

Using cURL with a Service Account for authentication, the following example removes the specified Service Accounts membership in the specified organization.

```bash
curl -X DELETE "https://api.d21s.com/v2/organizations/<ORGANIZATION_ID>/members/<SERVICE_ACCOUNT_ID>" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
```

{% endtab %}

{% tab title="Python API" %}
Once the package is installed and authenticated as described in the [Python API Reference](https://developer.disruptive-technologies.com/api/libraries/python/index.html#), an organization member can be removed by calling the following resource method.

* [disruptive.Organization.remove\_member()](https://developer.disruptive-technologies.com/api/libraries/python/resources/project.html#remove-member)

#### Example Usage

Using our Python API with Service Account credentials for authentication, the following example removes the specified Service Accounts membership in the specified organization.

```python
import disruptive as dt

# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)

# Remove member from the specified organization.
dt.Organization.remove_member(
    member_id='<MEMBER_ID>',
    organization_id='<ORGANIZATION_ID>',
)
```

{% endtab %}
{% endtabs %}

## Roles

A role contains permissions that determines which actions a member is authorized to perform on a specific resource in DT Cloud. To make permissions available to members, you grant them the role that provides the desired set of permissions.

The following roles are available.

* `project.user`
* `project.developer`
* `project.admin`
* `organization.admin`

### Managing Roles

Roles can be managed by users or Service Account Members with an assigned role of Project Administrator or Organization Administrator for projects and organization levels. The role of a member is managed in the same place where the member itself is managed. See the [Members](/service-accounts/managing-access-rights#members) section for details on where this is located, or use our REST API endpoint for Membership and Access Control.


# Permissions

Reference table over permissions granted per role.

The following table shows a complete list of permissions granted per role. You can read about how to grant the various roles on the [Managing Access Rights](/service-accounts/managing-access-rights) page.

* Permissions granted by the `organization.admin` implies elevated permissions for all projects, devices, and services within the organization it was granted for.

| Permission                  | <p>project<br>.user</p> | <p>project<br>.developer</p> | <p>project<br>.admin</p> | <p>organization<br>.admin</p> |
| --------------------------- | :---------------------: | :--------------------------: | :----------------------: | :---------------------------: |
| `dataconnector.create`      |                         |               X              |             X            |               X               |
| `dataconnector.read`        |            X            |               X              |             X            |               X               |
| `dataconnector.update`      |                         |               X              |             X            |               X               |
| `dataconnector.delete`      |                         |               X              |             X            |               X               |
| `device.read`               |            X            |               X              |             X            |               X               |
| `device.update`             |                         |               X              |             X            |               X               |
| `device.transfer`           |                         |                              |             X            |               X               |
| `emulator.create`           |                         |               X              |             X            |               X               |
| `emulator.read`             |            X            |               X              |             X            |               X               |
| `emulator.update`           |                         |               X              |             X            |               X               |
| `emulator.delete`           |                         |               X              |             X            |               X               |
| `membership.create`         |                         |                              |             X            |               X               |
| `membership.read`           |            X            |               X              |             X            |               X               |
| `membership.update`         |                         |                              |             X            |               X               |
| `membership.delete`         |                         |                              |             X            |               X               |
| `organization.read`         |                         |                              |                          |               X               |
| `organization.update`       |                         |                              |                          |               X               |
| `project.create`            |                         |                              |                          |               X               |
| `project.read`              |            X            |               X              |             X            |               X               |
| `project.update`            |                         |                              |             X            |               X               |
| `project.delete`            |                         |                              |             X            |               X               |
| `serviceaccount.create`     |                         |                              |             X            |               X               |
| `serviceaccount.read`       |            X            |               X              |             X            |               X               |
| `serviceaccount.update`     |                         |                              |             X            |               X               |
| `serviceaccount.delete`     |                         |                              |             X            |               X               |
| `serviceaccount.key.create` |                         |                              |             X            |               X               |
| `serviceaccount.key.read`   |            X            |               X              |             X            |               X               |
| `serviceaccount.key.delete` |                         |                              |             X            |               X               |


# Organizational Structures

A discussion of different methods of distributing access rights on an organizational level.

## Within a Single Organization

A single company often operates several sensors distributed between installations. These installations can be independent both in development and deployment, requiring consideration for how access rights are managed.

### Project-Isolated Access

Where different installations are separated by project, one method may be to manage the access rights within the project itself. By creating Service Accounts within projects as required, with different roles, only the required access needs to be granted.

This would allow all projects to be independent of each other regarding access rights. For a Service Account to access an installation, the credentials for that specific project would have to be obtained. This allows for much more granular control at the expense of some management.

![](/files/-MTb9HSFRVdHVm7jMtPH)

### Distributed Access

Similar behavior to [Project-Isolated Access](/service-accounts/organizational-structures#project-dependent-access) could also be achieved by having a single project dedicated to Service Account management.  Here, multiple Service Accounts would be created in a single project and given a role in different projects depending on the task they’re going to perform.

In terms of granularity of control, the result is much the same as having Service Accounts in the related projects. However, depending on how the task of managing the access rights falls within the organization, this method may prove to be simpler in generating an overview of who can access what.

![](/files/-MTb9KIijc49dlT4J8sJ)

## Between Organizations

In cases where Company A provides a service to Company B, an exchange of access rights may be necessary. Service Accounts can be used to accomplish this safely.

### An example approach

We have created a mock situation where Company A would like to provide a monitoring and alerting service to Company B and C.

* Company A creates a project with an accompanying Service Account for each of their clients, Company B and C.<br>
* The credentials for these Service Accounts are copied to the monitoring system, using the Service Accounts to fetch data through the REST API.<br>
* Company A provides Company B and C with the respective Service Accounts email addresses and requests to create a member with sufficient access. Company B gives project-level access rights as only a few projects are to be monitored. Company C gives organization-level access, allowing Company A to fetch data from all projects.<br>
* Afterward, Company A’s monitoring system can log in using the Service Accounts and list the projects and the sensors within them, then fetch data as they see fit. For Company C, which gave organization-level access, Company A can create and delete projects, move devices between all the projects, and invite new members to the organization.

![](/files/-MTbMp-GhMiOBGj5aX8Z)

This approach would allow both the service provider, Company A, and its customers to be in control. Company B and C can at any time revoke access to Company A in their respective projects by removing the Service Account members. Likewise, Company A is in a position where it can invoke security routines such as rotating key-ids at will.

### Simplified Approach

Company A could also use a single Service Accounts residing in its inventory project. This would simplify management as the same Service Account Email would be handed out to all clients. While simpler, this could make the probability of mixing clients higher, though it does not come at the expense of security. The members created by the clients fetch data in the monitoring service, keeping them isolated.

![](/files/-MTbMsuHnYTjUB12TN1n)


# Application Notes

Index page for the various application notes written to experiment with different sensor use-cases.

## Generating a Room Temperature Heatmap

Continuously logging temperature data in an office- or industry environment can provide a range of benefits. In this application note, a method of combining several Disruptive Technologies (DT) temperature- and door-proximity sensors to generate a temperature heatmap is proposed. Using a room layout representation, the heatmap gradients are calculated by inverse distance weighting. [Read more](/other/application-notes/generating-a-room-temperature-heatmap).

![](/files/-MUJMhQDE2G1dxzGrPy4)

## Modeling Fridge Content Temperatures

Continuously logging temperature data can help ensure the quality of refrigerated items in cold storage. However, in applications where the fridge door is opened in rapid succession, the unit's ambient air temperature may not reflect that of the items stored there. In this application note, a method for indirectly monitoring the core temperatures of items through heat-transfer modeling is explored. [Read more](/other/application-notes/modeling-fridge-content-temperatures).

![](/files/-MTewa2xs9e4H9N81q9k)

## Outlier Detection on Multiple Temperature Datastreams

When running large-scale services, continuously monitoring asset temperatures can provide essential information for smooth long-term operation. Due to their small size and long-lasting battery life, Disruptive Technologies (DT) Temperature Sensors are well suited for simultaneously monitoring many assets. In this application note, the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is applied on a stream of 25 temperature sensors with the aim of catching outlier events. [Read more](/other/application-notes/outlier-detection-on-multiple-temperature-datastreams).

![](/files/-MUJHtY2LLMeIjcmlvtm)

## Simple Temperature Forecasting for Substation Transformers

Forecasting time series data can be useful in many applications, be it simply for acquiring an overview of expected behavior or as incorporated in more complex anomaly detection systems. In this application note, the popular Holt-Winters model has been implemented and used for decomposing and forecasting seasonal transformer temperature data. Read more.

![](/files/-MUJRgWs2RI9hhy6nqnd)

## Sensor Data Insight with Power BI Integrations

Disruptive Technologies (DT) sensors can be used to gather high-resolution environmental data which in turn can be used to develop models and aggregate analysis. In this application note, we will look at how the POWER BI platform can be used to develop an analytics solution on data continuously gathered using DT sensors and a Data Connector. [Read more](/other/application-notes/sensor-data-insight-with-power-bi-integration).

![](/files/-MSDM0omhVW4eRNuEsdn)

## Third-Party Sensor Data in DT Cloud

While our current range of devices already covers many use cases, some sensor types, like barometric pressure, are yet not in our repertoire. In this application note, we will take a look at how a Raspberry Pi can be used to sample barometric pressure data from a Mikroe Environment Click breakout board, then periodically publish events to the DT Cloud using our Python Emulator API. [Read more](/other/application-notes/third-party-sensor-data-in-dt-studio).

![](/files/-Mcry19Hg5W-t2qSHxLa)


# Generating a Room Temperature Heatmap

## Introduction

Continuously logging temperature data in an office- or industry environment can provide a range of benefits. Be it optimizing energy usage by minimizing heat loss or tracking individual equipment temperatures, this information could be useful to gain better insight. Due to their small size, robustness, and long battery life, [Disruptive Technologies (DT) Wireless Temperature Sensors](https://support.disruptive-technologies.com/hc/en-us/articles/360010342900-Wireless-Temperature-Sensor) can easily be employed to collect temperature data streams for almost any environment.

In this application note, a method of combining several DT temperature- and door-proximity sensors to generate a temperature heatmap is proposed. By using room layout features, the heatmap gradients are calculated by inverse distance weighting (IDW) \[1]. The distances are found by euclidean path-finding \[2], where traversal can not occur through walls, only around corners and through doors. The result is a heatmap where rooms are separated by the walls between them but allows for temperature gradients to expand through doors and open spaces like the wave of a droplet.

![Figure 1: Temperature heatmap on a particularly hot day. Here, temperature sensors are represented by an 'x'. Doors can be in an open (green) or closed (red) state. An animation showing the temperature change during the entire day can be found at the end of this application note.](/files/-MUJMhQDE2G1dxzGrPy4)

## DT Studio Project Configuration

The implementation is built around using the developer API to interact with a single DT Studio project containing all sensors for which the heatmap is generated. If not already done, a project needs to be created and configured to enable API functionalities.

### Project Authentication

For authenticating the developer API against your DT Studio project, a Service Account key-id, secret, and email must be known, later to be used in the example code. If you're unfamiliar with the concept you can read our [Introduction to Service Accounts](/service-accounts/introduction-to-service-accounts).

### Adding sensors to the project

Any number and type of sensors can be included in the project. Only those included in the room layout file, which will be discussed later, are fetched by the API. The use of labels is not necessary for this implementation. The option to move sensors from one project to another can be found when viewing a particular sensor in DT Studio.

![Figure 2: DT Studio sensor overview menu where, among other things, sensors can be moved to different projects.](/files/-MUJO0TB7ir2BVCBOfGM)

## Sensor Placement

For the heatmap to produce an output consistent with the ambient temperature in the room independent of external sources, some thought should be put into sensor placement. During testing, it was found that a relatively large temperature difference could be seen depending on the height at which the sensor was placed, which was expected as warm air rises. The following recommendations should, therefore, be considered when placing the sensors.

* Avoid placing sensors in direct sunlight.
* Avoid placing sensors near air-conditioning, radiators, or other heat sources. This includes people, which are comparable to a 100-watt heat-source.
* Placement height should be somewhat consistent between sensors. Approximately chest-level height was found to provide the most representative results.

![Figure 3: Disruptive Technologies Temperature Sensors mounted on a wall.](/files/-MUJO9reasIFmO-HdH2z)

## Example Code

An example code repository is provided in this application note. It illustrates one way of building a heatmap from multiple sensor streams and could serve as a precursor for further development and implementation. It uses our [REST API](/introduction-to-rest-api) to interface with the DT Studio project and constructs a virtual representation of the environment for which details are provided by the user in a room layout JSON input argument.

### Source Access&#x20;

The example code source is publicly hosted on the official Disruptive Technologies GitHub account under the MIT license. It can be found by following [this link](https://github.com/disruptive-technologies/temperature-heatmap-an).

### Environment Setup

All code has been written in and tested for Python 3. While not required, it is recommended to use a virtual environment to avoid conflicts. Additional dependencies can be installed using pip and the provided requirements text file.

```
pip3 install -r requirements.txt 
```

Using the details found during the project authentication section, edit the following lines in *sensor\_stream.py* to authenticate the API with your DT Studio project.

```python
USERNAME   = "SERVICE_ACCOUNT_KEY"    # this is the key
PASSWORD   = "SERVICE_ACCOUNT_SECRET" # this is the secret
PROJECT_ID = "PROJECT_ID"             # this is the project id
```

### Defining the Rooms Layout

In order to provide the layout of your environment to the example code project, the JSON file format is used and should be provided as an input argument. If no layout is provided, the project will default to a sample layout located in *./config/sample\_layout.json*. Please see "Layout Construction" under the Implementation section for details about setting up your environment layout. It is recommended that the sample layout structure is understood before designing your own.

### Usage

If the example code is correctly authenticated to the DT Studio project as described above, running the script *sensor\_stream.py* will generate and show the sample layout.&#x20;

```
python3 sensor_stream.py 
```

If a custom layout file is provided, the script will start streaming data from all initialized sensors in the layout for which a heatmap is continuously updated as new data arrive. For more advanced usage, such as setting a historical time frame to pull data from, provide any of the following extra arguments and values.

```
usage: sensor_stream.py [-h] [--layout] [--starttime] [--endtime] [--timestep]
                        [--plot] [--debug] [--read]

Heatmap generation on Stream and Event History.

optional arguments:
  -h, --help    show this help message and exit
  --layout      Path to json layout file.
  --starttime   Event history UTC starttime [YYYY-MM-DDTHH:MM:SSZ].
  --endtime     Event history UTC endtime [YYYY-MM-DDTHH:MM:SSZ].
  --timestep    Heatmap update period.
  --plot        Plot the estimated desk occupancy.
  --debug       Disables multithreading for debug visualization.
  --read        Import cached distance maps. 
```

The arguments *--starttime* and *--endtime* should be of the format *YYYY-MM-DD**T**hh:mm:ss**Z***, where *YYYY* is the year, *MM* the month, and *DD* the day. Likewise, *hh*, *mm,* and *ss* are the hour, minutes, and seconds respectively. Notice the separator, ***T***, and ***Z,*** which must be included. It should also be noted that the time is given in UTC. Local timezone corrections should, therefore, be made accordingly.

With a default value of 1 hour, the --timestep argument represents the number of seconds between each time the heatmap is updated when looking at historical data. During a stream, the heatmap will update every new event regardless.

## Implementation Details

With the aim of generating a heatmap that continuously updates as new temperature data arrives in the stream, the implementation is written such that any processing-intensive task is done during initialization. By generating a Euclidean distance map for each sensor covering the entire room layout, the inverse distance weighted (IDW) temperature value at any point can quickly be calculated using the maps as lookup tables. Therefore, to update the heatmap, only a single scan of the grid is required. This results in a total delay of only a few seconds from sensor sampling heartbeat to heatmap update. The maps are written to temporary files and can be loaded on successive runs.&#x20;

### Layout Construction

To illustrate how custom room layouts are generated, the provided sample layout will be used as an example. It consists of 2 rooms, a door connecting them, and a few sensors with some initial temperature. The constructed sample layout is shown in figure 4, where sensors are represented by 'x', and objects of interest (OOF) by a circle. An OOF is simply a temperature value that is displayed without affecting the heatmap itself.

![Figure 4: Sample layout provided with the example code. Here, an 'x' represents a sensor, a green filled line an open door, and a circle an OOF.](/files/-MUJOiHQCpt8BVyi_6JB)

The following lines show the JSON format used to generate the room layout shown in figure 4. Each room in the room list is defined by a name, a list of corners, and the sensors that exist in the room. Observe how the corners are listed in a clockwise manner. Following this template, the layout can be expanded to an undefined number of rooms and sensors.

```javascript
{
    "rooms": [
        {
            "name": "room0",
            "corners": [
                {"x": 0.0, "y": 0.0},
                {"x": 0.0, "y": 5.0},
                {"x":10.0, "y": 5.0},
                {"x":10.0, "y": 0.0}
            ],
            "sensors": [
                {"x": 0.2, "y": 2.5, "sensor_id": "xxxxx", "t0": 23},
                {"x": 5.0, "y": 4.8, "sensor_id": "xxxxx", "t0": 22},
                {"x": 8.0, "y": 0.2, "sensor_id": "xxxxx", "t0": 24}
            ]
        },
        {
            "name": "room1",
            "corners": [
                {"x":10.0, "y": 0.0},
                {"x":10.0, "y":8.0},
                {"x":15.0, "y":8.0},
                {"x":15.0, "y": 0.0}
            ],
            "sensors": [
                {"x":12.5, "y": 7.5, "sensor_id": "xxxxx", "t0": 22}
            ]
        }
    ],
    "doors": [
        {
            "p1": {"x":10.0, "y": 3.0}, 
            "p2": {"x":10.0, "y": 4.0}, 
            "room1": "room0", 
            "room2": "room1", 
            "sensor_id": "xxxxx",
            "closed": false
        }
    ],
    "oofs": [
        {"x": 17.5, "y": 8.3, "sensor_id": "xxxxx"}
    ]
}
```

Sensors can be initialized with a temperature $$t0$$. This value is only used if the script does not connect to the API, such as when using the sample layout. Then, the heatmap will be generated using said initial values instead. These can, however, be omitted and are meant for debugging purposes. The same goes for initializing doors with a *closed* boolean.

Considerations when designing your layout:

* Sensors must not be placed exactly on walls. This will create a line-of-sight issue when calculating the euclidean distance map. Sensors should therefore be slightly offset towards the room center. Any positive non-zero value will suffice.
* The corners of a room must be provided in chronological order, as two adjacent corners define a single wall.&#x20;
* Two adjacent rooms should only share a single door. While a single room can have doors leading into multiple rooms, only a single door should be used between any one pair of rooms. This is to avoid generating an exponential amount of distance maps.

### Euclidean Distance Map Population

In order to produce a heatmap in which each sensor's contribution is weighted by their distance, the distance from each sensor to every point in the grid needs to be known. This would be rather simple if walls were not to be considered. However, in order to create a more believable heatmap, the distance from a sensor to any point in any room can only be calculated as a path through doors, and not walls, as is the case with airflow.

For every sensor, a distance map representing the shortest Euclidean distance to every point in the grid is created where the euclidean distance is given by

$$
d = \sqrt{\sum\_{i=1}^{n}(q\_i-p\_i)^2},
$$

where $$q\_i$$ and $$p\_i$$ are the $$i$$-th coordinate values of two points in an $$n$$-dimensional space. The following code snippet represents the recursive function implemented to do a breadth-first search for the shortest path from a sensor to any door- and convex corner \[3].

```python
def find_shortest_paths(current, path, path_length):
    # append path with currently active point
    path.append(current)

    # stop if we've been here before on a shorter path
    if path_length > length of current.shortest_path:
        return path

    # copy path to current point to save it
    current.shortest_path = path

    # find candidate corners for path expansion
    candidates = find(doors and convex corners in line of sight of current)

    # recursively iterate candidates
    for c in candidates:
        # calculate distance from current to candidate
        new_distance = euclidean_distance(current.x, current.y, c.x, c.y)

        # next recursive step
        path = find_shortest_paths(c, path, length of path + new_distance)
            
        # remove current step from path
        path.pop()
            
    return path
```

Essentially, for each recursive step starting at the sensor position, all convex corners and doors in line of sight are defined as candidates for the next step. Here, the line of sight is defined as a point to which a direct line can be stretched without intersecting a wall. Concave corners are ignored as they will never represent the shortest path. As they are visited, each point subsequently saves the shortest path to it, as shown in figure 6, used in the next step to populate the euclidean distance map.

![Figure 5: Shortest paths (red) from a sensor to all convex corners and doors in layout found through recursively locating all corners within line of sight at any step.](/files/-MUJP_fmHl5yKAq08ZuL)

For each step in every recursive path found previously, the euclidean distance to all points in the grid within its line of sight is calculated. If the new distance plus the total length traveled by the path is shorter than the existing value, it is updated. The result is a grid in which each point represents the shortest possible distance from the initial sensor location, shown in figure 6, where an increase in brightness represents a longer distance traveled. Notice how the distance gradient does not increase through walls but instead enters a room only through a doorway as expected.

![Figure 6: Euclidean distance grid population process where each point in the grid is filled with the shortest distance from the sensor, traveling through a path of convex corners and doors as necessary. Here, as distance increases, the color becomes lighter.](/files/-MUJPh-U64VugHBWQ3rh)

For every new temperature value that is received in the stream, the heatmap is recalculated. This is very quick as the distance from every sensor to every point is known from the previous steps. Therefore, the heatmap grid is scanned once, where for each grid point, the value is set to the inverse distance weighted (IDW) temperature of all sensors. The IDW is given by

$$
d = \sum\_{i=1}^{n}{\frac{z\_i}{d\_i^p}}/\sum\_{i=1}^{n}{\frac{1}{d\_i^p}},
$$

where $$n$$ is the number of sensors, $$z\_i$$ the value of sensor $$i$$, and $$d\_i^p$$ the distance to sensor $$i$$ to the power of $$p$$. It is a simple but flexible spatial interpolation method that produces a smooth gradient between sensors. Other methods, such as kriging statistical modeling or radial basis functions, could also be experimented with to produce the heatmap.

### Door State

To produce a more natural heatmap, the temperature gradient in a room is only affected by sensors in other rooms - should the door between them be open. By using [Proximity Sensors](https://support.disruptive-technologies.com/hc/en-us/articles/360010452099-Wireless-Proximity-Sensor) in the door frame, the heatmap can be automatically updated should doors open and/or close.

Due to calculating the euclidean distance map earlier, information regarding which doors the path pass through is known. Therefore, to determine whether or not a sensor should be involved in the IDW estimate for a given grid point during heatmap update, only a simple boolean check for the doors in each path has to be done with a negligible execution time impact. Figure 7 shows how the temperature gradient changes when a door between a cold and warm room opens (green) and closes (red).

![Figure 7: Difference in temperature gradient depending on whether or not the door between the two rooms are open (green) or closed (red).](/files/-MUJQUKtRsxMQtPfmJUV)

## Animated Result

The following animation represents the same day, as shown in figure 1, but includes the temperature variations for the whole day. A second in the animation represents 2 hours of real-time. Notice how doors open and close, and how the temperature spikes in the morning, before being brought back down to normal by the air-conditioning system kicking in.

![Figure 8: Animated heatmap for an entire day where the temperatures were particularly hot, showing how the air-conditioning system kicks in to suppress the heat.](/files/-MUJQ_ywntA7MPv_3V9p)

## Future Improvements

* Currently, walls can only be defined as a straight line between two points. Spline interpolation could be implemented to solve this should it be necessary.
* Doorways are currently implemented so that the temperature gradient only passes through the exact middle. This approximation reduces the recursive depth when finding the shortest path while having a negligible impact on the resulting heatmap. However, if a doorway is several meters wide, this would be noticeable and should be fixed.

## References

1. <https://en.wikipedia.org/wiki/Inverse_distance_weighting>
2. <https://en.wikipedia.org/wiki/Pathfinding>
3. <https://en.wikipedia.org/wiki/Breadth-first_search>&#x20;


# Modeling Fridge Content Temperatures

## Introduction

In cold storage applications, continuously logging temperature data can help ensure the quality of refrigerated items. Due to their small size and long-lasting battery, one- or several [Disruptive Technologies (DT) Wireless Temperature Sensors](https://www.disruptive-technologies.com/products/wireless-sensors) placed inside a fridge or freezer will provide high-resolution data for continuous monitoring and detection of anomalous events.

However, in applications where the fridge door is opened in rapid succession, the unit's ambient air temperature may not reflect that of the items stored there. While the air may see a quick rise, an item's thermal inertia will keep it chilled for quite some time. Therefore, if alarms are used to detect higher than wanted temperatures, alarm-fatigue is likely. One possible solution is to place the sensor in a proxy medium, such as water, to emulate the thermal inertia. However, this might be impractical and can be solved without the extra required hardware.

In this application note, a method for indirectly monitoring the temperatures of refrigerated items through heat-transfer modeling is explored. By actively measuring only the ambient fridge temperature, Newton's Law of Cooling is applied to provide an estimate of individual food items stored within, while requiring only a single initial condition parameter.

![](/files/-MTewa2xs9e4H9N81q9k)

## Theory

Like other scientific fields, thermodynamics can be approached in a simplified manner if some approximations are made. Here, when regarding an item resting inside the fridge, thermal displacement is assumed to occur mainly through conduction with the shelf and convection to circulating air \[1, 2]. This allows for the heat transfer coefficient to be regarded as a constant in each medium, reducing complexity. Also, differences in volume and surface area will, in reality, have an effect on the rate at which an item cools. However, this can be overlooked if one assumes that each storage quanta of a product is approximately equal.

### Newton's Law of Cooling

Considering the aforementioned approximations, the simple yet elegant Newton's Law of Cooling can be applied to model the heat transfer of an item in the fridge by only measuring the ambient temperature. Given by

$$
\frac{Q}{A} = -k(y-T\_{\text{amb}}),     \tag{1}
$$

where $$Q/A$$ is the rate of heat transfer out of the body per unit area in $$W/m^2$$, or heat flux, $$k$$ is a constant heat transfer coefficient, $$y$$ the current estimated temperature, and $$T\_{\text{amb}}$$ the ambient, the law states that the temperature of a body changes at a rate proportional to the difference in temperature between the body and its surroundings \[3]. When written as a transient response

$$
y = (y\_0 - T\_{\text{amb}})^{-kt} + T\_{\text{amb}},     \tag{2}
$$

where $$y\_0$$ is the initial temperature, it can be seen that the change in temperature decreases exponentially as time progresses. If the environmental temperature $$T\_{\text{amb}}$$ is considered constant, $$k$$ alone will affect the individual rate of change. Figure 2 shows a simulation for a synthetic fridge temperature for which the model is applied with several different values of $$k$$.

![Figure 2: Simulations of Newton's Law of Cooling for several values of the heat transfer coefficient  k , showing how the thermal inertia manifests itself as the ambient temperature quickly rises.](/files/-MR_kCxpSSWz6uYod4v7)

### Heat Transfer Coefficient

In thermodynamics, the heat transfer coefficient is defined as the proportionality constant between the heat flux and the driving force for heat flow \[4]. Given by

$$
k = \frac{Q}{A\Delta T},     \tag{3}
$$

where $$Q/A$$ is the same heat flux as in equation (1) and $$\Delta T$$ the temperature difference between the medium and surrounding environment, it quantifies how well heat is conducted through a medium.

Though there are many mathematical approaches to finding $$k$$, by actually measuring the temperature change in a medium and fitting the model to it, an empirical solution can also be found. If it is assumed that the environment or medium does not change significantly over time, this can provide a relatively good approximation.

## Testing the Model

An experiment was performed to test the model in a real-world application. Six different items were tested using a consumer-grade refrigerator as the environment. A volume of water, yogurt, oatmeal, an apple, banana, and a slice of meat was inserted with Disruptive Technologies Wireless Temperature Sensors for reference. The ambient air temperature was measured using the Ambient Range Extender Accessory. Compared to attaching a sensor on the wall, it provides a better representation of the air temperature.

A change in temperature was forced by keeping the refrigerator door open. After cooling overnight to achieve equilibrium, the door was kept open for 1 hour to evaluate the heat transfer curves. Though this is probably longer and not representative of ordinary circumstances, it allows us to observe each item's thermal inertia for which the model is matched.

For exploratory purposes, the sensors in this test used a 30-second heartbeat. This did, however, prove to be much redundant. With its 5.5-minute heartbeat, the [Wireless Temperature Sensor EN12830/330s](https://support.disruptive-technologies.com/hc/en-us/articles/360010452139-Wireless-Temperature-Sensor-EN12830-330s) would still collect multiple samples in the time it takes for refrigerated items to experience a change in temperature, making it suitable for this purpose.

![Figure 3: Food items inserted with Disruptive Technologies Wireless Temperature Sensors to record a reference temperature towards which the model is evaluated.](/files/-MR_kt2B_has2TFk6e4H)

When disregarding the API interfacing to download sensor data, the implementation is straightforward. When using a constant value for $$k$$, equation (1) can update the model in only a few lines of code. A pseudocode representation of the updating function is shown below.

```python
def update_model(ambient_temperature, last_temperature, k):
    # calculate dy/dt, the change in temperature
    change_in_temperature = -k*(last_temperature - ambient_temperature)
    
    # add to produce new model temperature value
    new_temperature = last_temperature + change_in_temperature
```

The test results are shown in figure 4. For each subplot, the ambient fridge temperature, reference food-item temperature, and modeled temperature are shown. It is clear that the model quite accurately represents the thermal change in all of the tested items. As expected, the ambient temperature increases rapidly when the door is opened at 07:50. While the exact rate of change differs between the items, they all show the expected thermal inertia behavior, which is captured for both heating and cooling by the model.

![Figure 4: The fridge ambient temperature, raw temperature data and modeled temperature for each food item in the test spanning a few hours. The fridge door is kept open for 1 hour to force a change in temperature.](/files/-MR_l9s-N3nUu3VX8GQy)

The heat transfer coefficients used for the results presented in figure 4 are given in Table 1. These were found by mapping a model to the real data. Note that due to the approximations made in this experiment, these values should be verified or found anew when implemented.

| Medium  | $$k$$ \[$$W/(m^2 \cdot K)$$] |
| ------- | ---------------------------- |
| Water   | 0.010                        |
| Yogurt  | 0.015                        |
| Oatmeal | 0.025                        |
| Apple   | 0.010                        |
| Banana  | 0.016                        |
| Meat    | 0.015                        |

An interesting observation is the temperature offset, which can be seen in some of the graphs. The expectation is that an item would cool until an equilibrium with the ambient temperature was reached, which is also what the model assumes. However, looking at especially the meat- and water items, their equilibrium was found to be a degree or so above the ambient. This is almost certainly caused by the fridge not having a perfect homogeneous temperature environment. As the ambient is measured in a single spot, this non-uniformity may explain the effect. Commercial-grade fridges might prove better in this regard.

## Practical Application

Depending on the desired complexity, one might want to use a single general model for all items in a fridge. Fortunately, while the test shows that the model can accurately estimate each type of food, the thermal change difference between them is slight enough that an average heat transfer coefficient could be used for simplicity.

Figure 5 shows all item temperatures superimposed onto the ambient, both with- and without correcting for the temperature offset. With a few exceptions, for instance, with the oatmeal reaching a higher maximum, it can be seen that the curves overlap rather well. Therefore, if some error is acceptable, one could model a wide variety of fridge content with a single general model, reducing complexity significantly.

![Figure 5: Each food item temperature data superimposed onto the fridge ambient temperature for comparison without any correction (left) and with correcting for the offset (right).](/files/-MR_lggpbVwt_6dAm8uy)


# Outlier Detection on Multiple Temperature Sensors

## Introduction

When running large-scale services, continuously monitoring asset temperatures can provide essential information for smooth long-term operation. Whether it is large office spaces, machinery in a production line, or server racks in a data center, multiple sensors are in some applications placed at once. If one or more sensors report temperature values deviating too far from the norm, preventative steps can be taken to avoid further degradation.

Due to their small size and long-lasting battery life, Disruptive Technologies (DT) Wireless Temperature Sensors are well suited for monitoring large amounts of assets in parallel. Employable in almost any environment, by measuring the temperature every 15 minutes, the data trend and behavior can be monitored and possible outliers can be caught in real-time.

In this application note, the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is applied on a stream of 25 temperature sensors with the aim of catching outlier events. As shown in figure 1, the data from most sensors are pretty similar in both level and trend. Occurrences of sudden spikes or level shifts caught by the algorithm are therefore considered to be outliers where appropriate action can be taken.

![Figure 1: One week of temperature data from 25 DT Wireless Temperature Sensors where outlier events in the data caught by the DBSCAN algorithm are highlighted for visibility. ](/files/-MUJHtY2LLMeIjcmlvtm)

## Sensor Placement

If the aim is to highlight outlier behavior in the temperature originating from a specific device or environment, certain considerations should be taken when mountain the sensors. For instance, if room temperatures throughout a building are the source of interest, sensors should be placed away from external heating sources such as air-conditioning or direct sunlight. Otherwise, the algorithm might classify said external intervention as an outlier, resulting in false alarms.

## DT Studio Project Configuration

The implementation is built around using the DT Developer API to interact with a single DT Studio project containing all temperature sensors for which outlier detection is performed. If not already done, a project needs to be created and configured to enable the API functionality.

### Project Authentication

For authenticating the developer API against a Service Account in your DT Studio project, three separate authentication details have to be located, later to be used in the example code. If you're unfamiliar with the concept, refer to our [Introduction to Service Accounts](/service-accounts/introduction-to-service-accounts).

### Devices

The script will use all temperature devices in the target project. Note that DBSCAN works better the more devices you include, preferably 10 or more.

## Example Code

An example code repository is provided in this application note. It illustrates one way of detecting outliers in multistream data and is meant to serve as a precursor for further development and implementation. It uses our [Python API](https://developer.disruptive-technologies.com/api/libraries/python/client/quickstart.html#client-quickstart) to interact with the DT Studio project.

### Source Access

The example code source is publicly hosted on the official Disruptive Technologies GitHub repository under the MIT license. It can be found by following [this link](https://github.com/disruptive-technologies/outlier-detection-an).&#x20;

### Environment Setup

The code has been written in and tested for Python 3.9+. Dependencies can be installed using pip and the provided requirements text file.

```
pip3 install -r requirements.txt
```

Using your authentication details, set the following environment variables.

{% code title="sensor\_stream.py" %}

```python
export DT_SERVICE_ACCOUNT_KEY_ID='<YOUR_SERVICE_ACCOUNT_KEY_ID>'
export DT_SERVICE_ACCOUNT_SECRET='<YOUR_SERVICE_ACCOUNT_SECRET>'
export DT_SERVICE_ACCOUNT_EMAIL='<YOUR_SERVICE_ACCOUNT_EMAIL>'
```

{% endcode %}

### Usage

If the example code is correctly authenticated to the DT Studio project as described above, running the script *main.py* will start streaming data from each desk sensor in the project for which outlier detection is performed as new data arrive.

```
python3 main.py
```

Use the `-h` flag to print additional flags available.

## Implementation Details

Classifying data for outlier detection is an ongoing research field that has seen many approaches over the years. Lately, machine learning techniques have been the new frontier in this area at the cost of complexity. In contrast, clustering techniques can be comparably simple while still providing good performance. In particular, the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm has been found to provide good performance with relatively little parameter tweaking \[1].

### Preprocessing

Depending on the application, time-series data are often feature-engineered before being applied to a classification scheme. However, each sample in a time series of length NN can also be considered a feature in an NN-dimensional space and be applied directly. This was, during testing, found to result in much better performance than by extracting mean, kurtosis, skew, and other typical time-series features for cluster input.

![Figure 3: Windowing of the most recent 24 hours of data that are uniformly resampled before providing it as an  N -dimensional input for the DBSCAN clustering algorithm.](/files/-MUJKj8kMf9Itl0-rS_-)

### DBSCAN

Compared to the likes of k-means clustering, DBSCAN does not require prior knowledge about the number of clusters in the data. It is also unsupervised, simplifying its use in many applications. One feature that makes it particularly useful for outlier detection is its notion of noise in the data. If a point does not fit in any cluster, it is classified as noise instead of the closest match. Figure 4 shows the result of applying DBSCAN on some synthetic data with two features. [This website](https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/) provides excellent animated visualizations of the clustering procedure.

![Figure 4: DBSCAN applied to data in 2 dimensions, identifying two individual clusters and noise.](/files/-MUJKtyG62Y1qFUhWoUk)

When grouping the features into clusters, DBSCAN uses a distance metric, here [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance), to determine if two or more points should be linked. For this, the two search parameters ϵϵ and pp must be given, where $$ϵ$$ is the search radius and pp the minimum number of points that can define a cluster. When scanning the dataset, each NN-dimensional point is classified as one out of three possible categories. A core point is defined as one that neighbors at least pp other points within a distance of $$ϵ$$. A border point is one that can be reached by a core point, but does not fulfill the requirement itself, marking the edge of a cluster. If a point is not reached by any core point, it is defined as noise. Figure 5 shows an example of how points are classified to form a cluster.

![Figure 5: Cluster generation procedure of the DBSCAN algorithm where the  ϵ  neighborhood is found for each point, classifying said point as either noise, border, or core.](/files/-MUJL1TBGdP8KJfy4ou_)

Finding a balance between generalized behavior and performance is one of the challenges when choosing $$ϵ$$ and $$p$$. Here, if we assume that an outlier does not correlate with other potential outliers, setting $$p=2$$ should result in said outliers being classified as noise by DBSCAN as there should be no other similar series. On the other hand, $$ϵ$$ dynamically recalculated on each call to compensate for changes in the data. By finding the average of every time series in a window, ϵ is calculated as the median Euclidean distance from each series to the average.

## Real-time Application

The script can be extended to work in real time by utilizing the `disruptive.Stream` module in our Python API. Below is a short visualization of how outlier classification can work in real time. The GIF is significantly sped up here.

![Figure 6: DBSCAN being continuously applied to 25 different temperature data streams in realtime as they arrive in the stream, highlighting outlier data that differentiates itself.](/files/-MUJLHUuggpFUGoFwEvN)

## References

1. <https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html>
2. <https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/>
3. <https://en.wikipedia.org/wiki/Euclidean_distance>


# Simple Temperature Forecasting for Substation Transformers

## Introduction

Forecasting time series data can be useful in many applications, be it simply for acquiring an overview of expected behavior or as incorporated in more complex anomaly detection systems. While machine-learning forecasting methods have been shown to produce excellent results in this regard, more traditional statistical modeling approaches are still very much competitive, providing the benefits of being simple yet effective, computationally inexpensive, and explainable \[1].&#x20;

In this application note, the popular Holt-Winters model has been implemented and used for decomposing and forecasting seasonal transformer temperature data. The following sections explain step-by-step how you can set this up for your own Disruptive Technologies (DT) Studio project for forecasting your own data.

![Figure 1: Transformer temperature data forecasted 5 days into the future.](/files/-MUJRgWs2RI9hhy6nqnd)

## DT Studio Project Configuration

The implementation is built around using the developer API to interact with a single DT Studio project containing all sensors for which temperature is forecasted. If not already done, a project needs to be created and configured to enable API functionalities.

### Project Authentication

For authenticating the REST API against the DT Studio project, a Service Account key-id, secret, and email must be known, later to be used in the example code. If you're unfamiliar with the concept, read our guide on [Creating a Service Account](/service-accounts/creating-a-service-account).

### Adding sensors to the project

By default, all temperature sensors in the project will be individually treated and forecasted for. The number of sensors used does not have to be configured beforehand and is scaled automatically. The option to move a sensor between projects can be found when selecting a sensor in a DT Studio project as shown in figure 2.

![Figure 2: Detailed overview of sensors in DT Studio project.](/files/-MUJTMg4wo6zC1io-o6m)

## Example Code

An example code is provided in this application note. It illustrates one way of building a transformer temperature forecasting system and is meant to serve as a precursor for further development and implementation. It uses the [REST API](/introduction-to-rest-api) to interface with the DT Studio project.

### Source Access&#x20;

The example code source is publicly hosted on the official Disruptive Technologies GitHub repository under the MIT license. It can be found by following [this link](https://github.com/disruptive-technologies/simple-forecast-an).

### Environment Setup

All code has been written in and tested for Python 3. While not required, it is recommended to use a virtual environment to avoid conflicts. Required dependencies can be installed using pip and the provided requirements text file.

```
pip3 install -r requirements.txt 
```

Using the details found during the project authentication section, edit the following lines in *sensor\_stream.py* to authenticate the API with your DT Studio project.

```python
USERNAME   = "SERVICE_ACCOUNT_KEY"    # this is the key
PASSWORD   = "SERVICE_ACCOUNT_SECRET" # this is the secret
PROJECT_ID = "PROJECT_ID"             # this is the project id
```

### Usage

If the example code is correctly authenticated to the DT Studio project as described above, simply running the script *sensor\_stream.py* will start streaming data from each sensor in the project for which a forecast is continuously calculated as new data arrive.

```
python3 sensor_stream.py 
```

For more advanced usage, such as visualizing the forecast, one or several flags can be provided upon execution.&#x20;

```
usage: sensor_stream.py [-h] [--path] [--starttime] [--endtime] [--plot]
                        [--plot-debug]

Temperature forecast on Stream and Event History.

optional arguments:
  -h, --help    show this help message and exit
  --path        Absolute path to local .csv file.
  --starttime   Event history UTC starttime [YYYY-MM-DDTHH:MM:SSZ].
  --endtime     Event history UTC endtime [YYYY-MM-DDTHH:MM:SSZ].
  --plot        Plot the resulting forecast.
  --plot-debug  Plot algorithm operation. 
```

By providing the *--path* argument followed by the absolute path to a .csv file, the data contained in the said file will be loaded and forecasted upon. A data stream will not be started if used. The local .csv file must contain the columns *unix\_time* and *temperature*.

The arguments *--starttime* and *--endtime* should be of the format *YYYY-MM-DD**T**hh:mm:ss**Z***, where *YYYY* is the year, *MM* the month, and *DD* the day. Likewise, *hh*, *mm* and *ss* is the hour, minutes, and seconds respectively. Notice the separator, ***T***, and ***Z*** which must be included. It should also be noted that the time is given in UTC. Local timezone corrections should, therefore, be made accordingly.

By providing the *--plot* argument, a visualization of the latest forecast will be generated when available. If historic data is included, an interactive plot will be produced after estimating occupancy for the historic data. By closing this plot, the stream will start and a non-interactive plot will update for each sample that arrives.

Similar to *--plot*, the *--plot-debug* argument will also generate a visualization but of the detailed algorithm output. They work best when used on historic or local data and are implemented for debugging purposes.

## Implementation

With the aim of forecasting the temperature of a substation transformer several days into the future, the well-established Holt-Winters model has been implemented for real-time data collected with Disruptive Technologies Wireless Temperature Sensors. If configured properly, the model can forecast seasonality with an accuracy on par with more complex implementations such as SARIMA and LSTM \[1].

Though simple, the implementation does require some prior knowledge about the data on which it is applied. The expected seasonal sample period, i.e. hourly, daily, weekly, must be provided. If unknown, it is recommended that the model is changed for double exponential smoothing which does not require this information. Additionally,  a configurable number of seasons is required to initialize the model components. This is done to avoid initial erratic behavior.

The implementation has been structured such that typical tuning parameters for the algorithm are located in the file *./config/parameters.py.* As no single configuration can work for all data, users are encouraged to experiment with different combinations better suited for their own data.

![Figure 3: Algorithm flow chart from new event sample to forecast values.](/files/-MUJTgt9AFgMdN35bQrk)

### Holt-Winters Additive Method

Also known as Triple Exponential Smoothing, the Holt-Winters Method is an extension of the earlier Holt's Linear Trend Model \[2]. Through the addition of seasonality to the initial level- and trend components, the model has been a popular choice for forecasting and decomposition of time series. Though several variants of the model exist, such as the multiplicative method, the additive method represents data as a sum of three smoothing equations. With one equation per component, the isolated trend, level, and season of a signal $$y\_t$$ with a seasonal period $$m$$ are given by

$$l\_t = \alpha(y\_t-s\_{t-m}) + (1-\alpha)(l\_{t-1}+b\_{t-1})$$\
$$b\_t = \beta(l\_t - l\_{t-1}) + (1-\beta)b\_{t-1}$$\
$$s\_t = \gamma(y\_t - l\_{t-1} - b\_{t-1}) + (1-\gamma)s\_{t-m}$$

respectively, where $$α$$, $$β$$, and $$γ$$ are smoothing parameters tuned for individual datasets. Though usually found empirically through trial and error, one can apply optimization techniques such as steepest descent to find the optimum smoothing parameters. However, this has the potential of fitting a local minimum and does therefore still require some kind of manual intervention. Figure 4 shows a temperature time-series decomposed by the additive method.

![Figure 4: Temperature data and its isolated level, trend, and a seasonal component. ](/files/-MUJWAlj6zlT2uqDjPZX)

### Initialization

Due to the recursive nature of exponential smoothing, the level, trend, and seasonal components must be initialized with some values. While many approaches have been proposed, here we implemented one proposed by Rob Hyndman \[4]. A more detailed explanation can be found [here](https://robjhyndman.com/hyndsight/hw-initialization/).

In essence, a $$K$$-season moving average is fit to an initial length $$L\_0>K\cdot m$$ amount of temperature data, $$y\_0$$. After subtracting the moving average, the initial season $$s\_0$$ is defined as the average seasonal period in $$L\_0$$. By then expanding $$s\_0$$ over $$L\_0$$, it is subtracted from $$y\_0$$ for which linear regression is subsequently fit. The initial level $$l\_0$$ is defined as the regression intercept and initial trend $$b\_0$$ as the slope.

### Forecast

After initialization, an estimate of future values can be calculated using the level, trend, and season components. A $$h$$-sample forecast is given by

$$
\hat{y}*{t+h|t} = l\_t + hb\_t + s*{t+h-m(k+1)},
$$

where $$m$$ is the season length as previously mentioned. Essentially, this extends the exponentially weighted slope into the future from the given level, adding the seasonal component on top of it. Depending on the chosen smoothing parameters, the forecast may or may not capture small changes in the signal.

![Figure 5: Five day forecast of transformer temperature data.](/files/-MUJX0J8kKL0aKOsUsAN)

### Confidence Interval

Forecast values should be interpreted as an indicator of how the signal might behave in the future based on previous behavior, not as a guaranteed truth. This becomes more evident if a confidence interval is included, a range for which we expect $$y\_t$$ to land with a given percentage of certainty. The width is dependent on the accuracy of earlier forecasts, while also becoming wider the longer into the future we forecast. Therefore, for a hh-step forecast of our seasonal data, the confidence interval is given by

$$
\hat{d}\_{T+h|T}=c\hat{\sigma}=c\hat{\sigma}\sqrt{\frac{h-1}{m}+1}
$$

where $$\hat{\sigma}$$ is the standard deviation of previous forecast residuals and $$c$$ a multiplier. Assuming a normal distribution, $$c$$ can be found through a table lookup, where for instance choosing $$c=1.96$$ results in a $$95%$$ confidence interval \[3]. Figure 6 shows how the confidence interval expands for forecasts farther into the future.

![Figure 6: Confidence interval of five-day forecast.](/files/-MUJXbog7CW_2UOV-rzH)

## References

1. Makridakis S, Spiliotis E, Assimakopoulos V (2018) Statistical and Machine Learning forecasting methods: Concerns and ways forward. PLoS ONE 13(3): e0194889. <https://doi.org/10.1371/journal.pone.0194889>
2. <https://otexts.com/fpp2/holt-winters.html>
3. <https://otexts.com/fpp2/prediction-intervals.html>
4. <https://robjhyndman.com/hyndsight/hw-initialization/>


# Sensor Data Insight with Power BI and Azure

## Introduction

By utilizing [Disruptive Technologies (DT) Wireless Sensors](https://www.disruptive-technologies.com/products/wireless-sensors), collecting high-resolution data from almost any source is simple. With their long-lasting battery life, valuable information regarding the environment and assets can be continuously monitored in real-time. Over time, analyzing and modeling the aggregated data allows for changes in trend and behavior to be evaluated and reacted to. Combined, this information can be invaluable for any company looking to optimize their process, be it reducing production line downtime or increasing employee satisfaction.

Developed by Microsoft, POWER BI aims to connect and visualize data in one unified and scalable platform. Due to its powerful analytics and data exploration toolbox, many companies have adopted the platform for performance monitoring and decision-making support. Whether your company is already using POWER BI or wants to start exploring it now, the DT Sensor Solution makes it easy to integrate your sensor data to gain additional insight into daily operations.

In this application note, we will look at how Data Connectors can be used to forward data from [DT Cloud](https://www.disruptive-technologies.com/products/dt-cloud) to your own external database in real-time, where POWER BI can continuously query the data into your analytics solution. The integrating steps will be explained in detail without the need for prior programming knowledge. After reading, the only remaining step is to explore the many benefits continuous monitoring can have for you and your business.

![](/files/-MSDM0omhVW4eRNuEsdn)

## Integration Overview

Sensor data can be accessed through either the available [REST API](https://support.disruptive-technologies.com/hc/en-us/categories/360002378059-Developers) or by using the Data Connectors available in DT Studio. For customers who want to forward the data to an external storage or processing platform, these Data Connectors provide a simple but secure method of doing so. They are implemented as webhooks with additional redundancy and security measures to ensure your data reaches its destination as intended. In this integration, a Data Connector will forward data in real-time to an SQL Database hosted at Microsoft Azure.

Any popular cloud hosting services such as [AWS](https://aws.amazon.com/), [Google Cloud](https://cloud.google.com/), [IBM Cloud](https://www.ibm.com/cloud), or [Azure](https://azure.microsoft.com/en-us/) could be used in this integration. However, Azure provides the simplest setup due to the shared user account between it and POWER BI. By setting up an HTTP trigger [Azure Function](https://docs.microsoft.com/en-us/azure/azure-functions/functions-overview) that listens for messages from the Data Connector, the data can be forwarded to and stored in an SQL Database on the same platform in real-time. Then, querying the database from POWER BI only requires a few clicks of the mouse.

![](/files/-MSDM7z12kSQrB6f4RJ3)

## Before You Begin

This integration requires no previous knowledge regarding programming or cloud integration, and we will deploy using only the GUI. However, it does require some copy-and-pasting of pre-written code, and for this, all the necessary files have been provided in a [Supporting File Repository](https://github.com/disruptive-technologies/powerbi-integration-an). Additionally, all software and platforms used do, at the time of writing, provide trial periods. Assuming you own DT sensors, the integration can therefore be recreated for free.

### Requirements

* DT Studio Developer Access or higher.
* A Microsoft Azure Account. [This can be created for free](https://azure.microsoft.com/en-us/).
* Download and install [POWER BI Desktop](https://powerbi.microsoft.com/en-us/desktop/).

### Supporting File Repository

To avoid endless code preview sections or appendices, a repository containing all code and files used in this project is provided. By following [this link](https://github.com/disruptive-technologies/powerbi-integration-an), you can either download or view in your browser the code necessary for later steps.

When the following integration steps mention a path in the form of **./example/path/to/file.ext***,* this refers to a file in the provided repository. You are then expected to either copy, upload, or apply the said file's contents depending on usage.

## Forward Data to Azure

Before POWER BI can query data, it needs to be forwarded to an external storage location, here an Azure SQL Database. We will initialize a new empty database, the HTTP trigger function, and a new DT Studio Data Connector in the following steps. Then we will link all three elements and verify successful data transfer.

### 1 - Azure SQL Database

1.1 - If you do not already have one set up, follow [Microsoft's own guide on creating an Azure SQL Database](https://docs.microsoft.com/en-us/azure/azure-sql/database/single-database-create-quickstart?tabs=azure-portal). Be sure to note the **Server SQL admin username and password,** as this will be used several times later.

1.2 - After the database has been initialized, enter it and locate the **Query Editor**. Log in with the **Server SQL admin username and password** from step 1.1.

1.3 - Paste the content of **./azure/query\_editor.sql** and click on **Run**. This will generate the database tables needed to store the data from all of DT's sensors. Verify that the tables were generated by viewing the **Tables** tab.

![](/files/-MSDMn1OsAgthc0dvI7G)

1.4 - Under the **Connection Strings** tab, copy the contents of **ADO.NET** into a text editor of choice. Paste the text, and replace **{your\_password}** with the **SQL admin password** found in step 1.1. Do not include the brackets. Save this for later.

![](/files/-MSDMs011uLYL4EqlW1t)

1.5 - Under Overview, copy the **Server Name** text and save it for later. This will be used by POWER BI to query the database for data.

![](/files/-MSDMwWICm8CgFIiQg0J)

### 2 - HTTP Trigger Function

2.1 - Create a new Function App by using [Microsoft's own guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-azure-function#create-a-function-app) as a reference. The following parameters should be specifically set.

* **Runtime Stack:** .NET Core
* **Version:** Latest. \
  The remaining parameters can be left as default or set as desired.

2.2 - After the Function App has been initialized, click **Go To Resource**.

2.3 - Under the Configurations tab -> Applications, click "+New Application Setting" to add the following environment variables.\
**DT\_SIGNATURE\_SECRET:** A secret signature used to validate the requests coming from the Data Connector. Use a strong password/secret here. Save it for later, as we will use the same secret when creating the Cata Connector.\
**SQLDB\_CONNECTION:** Paste the SQL **Connection String** found in step 1.4.

![](/files/-MSDNHjs1Vp42U8zdvP_)

2.4 - Under the Configurations tab -> Function Runtime Settings, set the Runtime Version as \~2. Remember to click **Save**.

![](/files/-MSDNNk0mAuQ40ZfoAqi)

2.5 - Under the **Functions** tab, create a new function by clicking +Add. Choose the **HTTP trigger** template, and give it a name, like HttpTrigger1.

![](/files/-MSDNSPVN1AurKWvBQEt)

2.6 - Enter the function. Under the **Code + Test** tab, paste the code located in **./azure/httptrigger/httptrigger.csx**. This will extract the different data fields from the request body JSON received by the trigger, then, based on the event type, insert the data into the SQL database. Remember to click **Save**.

2.7 - In the same window, click the small icon to the **right of Test/Run** and copy the **Function URL**. Save this for later.

![](/files/-MSDNWsZ5y4etuF08LCq)

### 3 - DT Studio Data Connector

3.1 - In DT Studio, under **API Integrations**, you will find the Data Connectors tab. Here, create a new connector.

![](/files/-MSDNgggqeN0zZJQaLtq)

3.2 - Set the following parameters:\
**Data Connector Display Name:** Something unique.\
**Endpoint URL:** Paste the Function URL of the HTTP trigger found in step 2.7.\
**Signature Secret:** Must be the same value as **DT\_SIGNATURE\_SECRET** in step 2.3.\
\
By default, the Data Connector will forward all event types. You can, however, choose to only forward one or a few types if you so wish. Remember to **Save**.

### 4 - Verify Connection

4.1 - If all the previous steps work as intended, the **Success** counter should be iterated for each new event. You can either wait for one of your sensors to send a new value, tap a sensor to send a touch event, or use the [sensor emulator](https://support.disruptive-technologies.com/hc/en-us/articles/360013666800-Using-the-Sensor-Emulator).

You can, and should, also verify that data appears in your database. You can do this by in your Azure SQL Database right-clicking one of the tables created in step 1.3, then click **Select Top 1000 Rows**. Your data should be listed below.

![](/files/-MSDNrkr4EHaUlFAWn6A)

4.2 - If the Data Connector shows errors or the data does not show up in the SQL Database, check the function logs. In the same windows as for step 2.7, click **Logs** at the bottom of the screen. This will, for each new event, print information that can help you locate the problem.

## Power BI Setup

Now that the data are safely being forwarded to our SQL database, POWER BI can query the data for further processing and analysis. Assuming successful data forwarding, these remaining steps are less involved and is probably familiar to users previously experienced with the platform.

### 5 - Connect Power Bi to Your Azure Sql Database

5.1 - In your POWER BI Project, add a new data source by clicking **Get Data**.

![](/files/-MSDOBMNEX_b1chu_sWz)

5.2 - Choose **SQL Server Database**, and click **Connect**.

![](/files/-MSDOFz0hUEeEoBm3qpU)

5.3 - In the **Server** field, paste the **Server Name** string found in step 1.5. Click **OK**. If you're logged in to POWER BI with the same Microsoft account used in Azure, authentication should not be necessary. However, if you're prompted, use the **admin username and password** found in step 1.2.

5.4 - In the following Navigator, all your Azure databases are listed. Choose one or several of the tables we created earlier. We will here only focus on temperature data. Click **Transform Data**.

![](/files/-MSDONA6LmTlVFxAlpTO)

5.5 - For POWER BI to correctly read the timestamps, we need to convert the type from Text to Date/Time. For each imported table, right-click the **timestamp** column, then **Change Type** and **Date/Time**. Click **Close & Apply** to transform and import the data.

![](/files/-MSDORDglwH4uR4cnK0k)

## Example Application

To test our integration, the implementations discussed in the two previously written application notes, [Modeling Fridge Content Temperatures](https://support.disruptive-technologies.com/hc/en-us/articles/360016431480-Modeling-Fridge-Content-Temperatures) and [Simple Temperature Forecasting for Substation Transformers](https://support.disruptive-technologies.com/hc/en-us/articles/360015485620-Simple-Temperature-Forecasting-for-Substation-Transformers), have been attempted recreated in a POWER BI Dashboard. Some additional statistics have also been added, mostly for exploratory purposes.

However, temperature forecasting can be applied simply by utilizing the built-in analytic found in POWER BI. The inertia model is implemented by utilizing the Python Visual element, which lets you transform the data through a custom Python script. It is available in **./powerbi/temperature\_inertia.py** should you wish to model your own temperature data inertia, assuming you have a working Python environment on your machine.

![](/files/-MSDOXz7TDpZyrTSAoZu)


# Third-Party Sensor Data in DT Cloud

## Introduction

When using Disruptive Technologies sensors, your data is continuously captured and forwarded by a Cloud Connector to the DT Cloud where it is stored for 31 days. While our current [range of devices](/concepts/devices) already covers many use cases, some sensor types, like barometric pressure, are yet not in our repertoire. However, while we continue to expand our offerings, a method of including third-party sensor data into our cloud does exist.

In this application note, we will take a look at how a [Raspberry Pi](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) can be used to sample barometric pressure data from a [Mikroe Environment Click](https://www.mikroe.com/environment-click) breakout board, then periodically publish events to the DT Cloud using our Python [Emulator API](/sensor-emulator). This lets the third-party pressure data propagate our cloud just like any first-party sensor, allowing you to effortlessly expand your integrations.

![](/files/-Mcry19Hg5W-t2qSHxLa)

## Prerequisites

* **Development Platform**\
  A [Raspberry Pi](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) or any other development board that can connect to our REST API. It is assumed that you are at least familiar with your platform of choice.
* **Third-Party Sensor**\
  An [Environment Click](https://www.mikroe.com/environment-click) or any other third-party sensor that can be sampled by your platform.
* **Service Account**\
  Creating, deleting, and interacting with emulated devices require that your User or [Service Account](/service-accounts/introduction-to-service-accounts) has been granted the role of Project Developer or higher.

## DT Studio Configuration

Only minimal [DT Studio](https://studio.disruptive-technologies.com/) setup is required to recreate the steps in this application note.

1. Navigate to the DT Studio project into which to forward your third-party sensor data.
2. Make sure you have a Service Account with a role of Project Developer or higher that is granted membership in the project.
3. For each third-party sensor data stream, you wish to forward, [Create An Emulated Sensor](/sensor-emulator#creating-an-emulated-device). We will use the Temperature Sensor type as a proxy to our data, even though we sample pressure data.
4. Take note of both the sensor- and project IDs as these will be used for publishing the events.

![](/files/-Mcrw9SJM8imdSQSHnOM)

## Implementation

This application note uses a Raspberry Pi with an Environment Click breakout board, but the implementation would be the same for any other platform. However, the GPIO pins and Python environment makes the Pi a good choice for quick and easy prototyping.

### Raspberry Pi 3 Model A+

It is assumed that your Pi is set up with the latest version of [Raspberry Pi OS](https://www.raspberrypi.org/software/) and has a working internet connection. If you're unfamiliar with this process, you should read the [official introductory guide](https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up).

#### Enabling I2C

We'll be using the I2C communication protocol to exchange information between the Pi and Environment Click breakout board. By default, this is disabled on the Pi, but can be enabled through the shell.

* `sudo raspi-config` -> Interface Options > I2C > Enable

#### Python Environment

We'll use the [Disruptive Python API](https://developer.disruptive-technologies.com/api/libraries/python/) to publish emulated sensor events with only a few lines of code. Therefore, a working Python 3 environment must be configured.

1. Install Python 3.7 or greater, either through the use of [Pyenv](https://github.com/pyenv/pyenv) or the official repository with `sudo apt install python3 python3-pip`.
2. Install the Disruptive Python API package with `pip3 install disruptive`.
3. Additionally, install any device driver packages to interact with your sensor. For our Environment Click board which hosts the [Bosch bme680](https://cdn-shop.adafruit.com/product-files/3660/BME680.pdf) sensor, [Adafruit](https://github.com/adafruit/Adafruit_CircuitPython_BME680) has a readily available package we can use `pip3 install adafruit-circuitpython-bme680`.

### Mikroe Environment Click

The Environment Click requires no configuration other than connecting it to the Pi.

* As we're using I2C, connect the SDA and SCL pins between both devices.
* Power the breakout board using the Pi 3V3 and GND pins.

![](/files/-McPyl-vbSCstH5redcR)

### Sample and Publish Events

Once you have a working Python 3 environment and the breakout board is connected, the sensor can be sampled by the Pi, then published as an event to the DT Cloud. The following snippet does exactly this.

1. Authenticate the [Disruptive Python API](https://developer.disruptive-technologies.com/api/libraries/python/) package using your Service Account credentials.
2. Initialize an instance of the bme680 sensor which the Pi will sample every 1 second.
3. For each new pressure sample, append a buffer list for storage between publishing events.
4. Once every 60 seconds, publish the median pressure value to an emulated device.

Remember to replace variables on the form `<VARIABLE>` with your own details.

```python
import time
import statistics

import adafruit_bme680
import board
import disruptive as dt

# Publishing-related constants.
PUBLISH_INTERVAL = 60  # seconds
PROJECT_ID = '<YOUR_PROJECT_ID>'
DEVICE_ID = '<YOUR_EMULATED_DEVICE_ID>'

# Authenticate the disruptive package.
dt.default_auth = dt.Auth.service_account(
    key_id='YOUR_SERVICE_ACCOUNT_KEY_ID>',
    secret='YOUR_SERVICE_ACCOUNT_SECRET>',
    email='<YOUR_SERVICE_ACCOUNT_EMAIL>',
)

# Create sensor object, communicating over the board's default I2C bus.
i2c = board.I2C()
bme680 = adafruit_bme680.Adafruit_BME680_I2C(i2c)

# Set sea-level barometric pressure to match your location.
bme680.sea_level_pressure = 1006  # hPa

# Initialize a buffer list to hold values between publishing.
pressures = []

# Start a timer for tracking publishing interval.
t0 = time.time()

# Suspend in an infinite loop, exiting at KeyboardInterrupt.
try:
    while True:
        # Read and append barometric pressure reading from bme680.
        pressures.append(bme680.pressure)

        # Print new pressure value to stdout.
        print(f'Pressure: {pressures[-1]:.2f}')

        # Check timer interval.
        if time.time() - t0 > PUBLISH_INTERVAL:
            # Have pressure be the median buffer value.
            pressure = statistics.median(pressures)

            # Publish a new event for the emulated device.
            dt.Emulator.publish_event(
                device_id=DEVICE_ID,
                project_id=PROJECT_ID,
                data=dt.events.Temperature(pressure),
            )

            # Reset timer, clear list, and tell stdout what you did.
            t0 = time.time()
            pressures = []
            print(f'Published event with value {pressure:.2f}.')

        # Sleep for a second.
        time.sleep(1)

except KeyboardInterrupt:
    pass
```

You probably notice that we use the [temperature event](/concepts/events#temperature-event) to publish our data. This is because a pressure event type does not exist, and since the data type is simply a floating-point number, it does not matter.

## Viewing Your Data in DT Studio

Once your script is running on your Raspberry Pi, head over to [DT Studio](https://studio.disruptive-technologies.com/). Here you should see the sensor data updating at each publishing interval. Using the Dashboard, a card can be created to inspect how your data behaves over time. &#x20;

![](/files/-Mci2f-AImRCt5wkwnqr)

Again, our cloud thinks the data is from a temperature sensor, but this is only a visual side-effect. If you choose to interact with- or forward the data it is, of course, unaffected.

## Next Steps

Now that your third-party sensor data is in our cloud, all services that are available to our first-party devices can be utilized. Here are a few examples of what you can do with your data.

* [Examples of how to integrate with various cloud services](/data-connectors/example-integrations)
* [Building a real-time heatmap](/other/application-notes/generating-a-room-temperature-heatmap)
* [Outlier detection on multiple data streams](/other/application-notes/outlier-detection-on-multiple-temperature-datastreams)


# Frequently Asked Question

## Data Connectors

#### Do you have an IP-range that Data Connectors will send events from?

We currently do not have a set IP-range. A Data Connector request origin can be verified by using Signature Secrets. You can read more about Signature Secrets on the [Advanced Configurations](/data-connectors/advanced-configurations) page.

#### My Data Connector returns errors. What do I do?

* Verify that your Data Connector is configured with the correct Endpoint URL.
* Check your endpoint logs for tracebacks that explain the source of the error.

#### Why did my Data Connector get automatically disabled?

A Data Connector can be automatically disabled if only non-200 responses have been returned for a long time. You can read more about this on the [Advanced Configurations](/data-connectors/advanced-configurations#enable-and-disable) page.

## REST API

#### What is the difference between the Event Stream API and a Data Connector?

[Data Connectors](/data-connectors/introduction-to-data-connector) are best used in backend integrations where a reliable and secure stream of events is desired, like forwarding events to external long-term storage. It comes with quality-of-life features such as an at-least-once Guarantee, low latency, and automatic scaling.

On the other hand, the [REST API Events Stream](https://developer.disruptive-technologies.com/api#/Live%20events%20streaming%20API/get_projects__project__devices_stream) provides a service aimed more at front-end applications and trades some quality-of-life features for flexibility. Granted the right permissions are given, a stream can be set up anywhere but does not come with the same at-least-once guarantee as a Data Connector.

## DT Cloud

#### How long are my sensor data stored?

All Events are stored for 30 days by default. For long-term storage, we recommend either using our managed [Extended Data Storage](https://support.disruptive-technologies.com/hc/en-us/articles/10488019175068-Extended-Data-Storage) or using a Data Connector to forward events to an external database that you manage yourself. You can get started with our [Example Integrations](/data-connectors/example-integrations).