Heroku

An example integration on forwarding Data Connector events to a server hosted on Heroku.

Overview

This example uses a Data Connector to forward the events of all devices in a project to a server hosted on Heroku. When receiving the HTTPS POST request, our application will verify both the origin and content of the request using a Signature Secret.

Prerequisites

The following points are assumed.

• You have the role of Project Developer or higher in your DT Studio project.

• You are familiar with the Introduction to Data Connectors and know how to Create a Data Connector.

• You are familiar with the Heroku Developer Documentation.

Heroku

Ensure you have the following software installed on your local machine.

• Git, the popular version control tool.

• Heroku CLI, for interacting with Heroku.

• A functioning local environment for the language of your choice.

Create a New App Locally

On your machine, create and enter a new directory for your app.

mkdir my-app
cd my-app

Python 3.11

In the directory, create a new file app.py where you paste the following snippet which contains the Flask server code for receiving and validating the Data Connector request.

import os
import hashlib

import jwt
from flask import Flask, request

app = Flask(__name__)

# Fetch environment variables.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')


def verify_request(body, token):
    # Decode the token using signature secret.
    try:
        payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
    except Exception as err:
        print(err)
        return False

    # Verify the request body checksum.
    m = hashlib.sha1()
    m.update(body)
    checksum = m.digest().hex()
    if payload["checksum"] != checksum:
        print('Checksum Mismatch')
        return False

    return True


@app.route('/', methods=['POST'])
def endpoint():
    # Extract necessary request information.
    body = request.data
    token = request.headers['x-dt-signature']

    # Validate request origin and content integrity.
    if not verify_request(body, token):
        return ('Could not verify request.', 400)

    # Print the request body.
    print(body)

    return ('OK', 200)

In the same directory, create a new file requirements.txt with the following contents. Heroku will install these dependencies in its environment before starting the server.

requirements.txt

gunicorn==20.1.0
flask==2.3.2
pyjwt==2.7.0

Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where Gunicorn should be used to serve the app using a variable called app in a module called app (app.py)

Procfile

web: gunicorn app:app

Python API

In the directory, create a new file app.py where you paste the following snippet which contains the Flask server code for receiving, validating, and decoding the Data Connector request.

app.py

import os

from dtintegrations import data_connector, provider
from flask import Flask, request

app = Flask(__name__)


@app.route('/', methods=['POST'])
def endpoint():
    # Use the provider-specific validation function.
    payload = data_connector.HttpPush.from_provider(
        request=request,
        provider=provider.FLASK,
        secret=os.getenv('DT_SIGNATURE_SECRET'),
    )

    # Print the payload data.
    print(payload)

    # If all is well, return 200 response.
    return 'Success'

In the same directory, create a new file requirements.txt with the following contents. Heroku will install these dependencies in its environment before starting the server.

requirements.txt

gunicorn==20.1.0
flask==2.3.2
dtintegrations==0.5.1

Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where Gunicorn should be used to serve the app using a variable called app in a module called app (app.py)

Procfile

web: gunicorn app:app

Node.js 16

In the directory, initialize a node application with the following command. This generates the file package.json which contains information used by Heroku when deploying the app.

npm init

Create a new file index.js where you paste the following snippet which contains the Express server code for receiving and validating the Data Connector request.

const express = require("express")
const bodyParser = require("body-parser")
const crypto = require('crypto')
const jwt = require('jsonwebtoken')

// Create the express server.
const app = express()
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())

// Fetch environment variables.
const signatureSecret = process.env.DT_SIGNATURE_SECRET

function verifyRequest(body, token) {
    // Decode the token using signature secret.
    let decoded;
    try {
        decoded = jwt.verify(token, signatureSecret)
    } catch(err) {
        console.log(err)
        return false
    }

    // Verify the request body checksum.
    let shasum = crypto.createHash('sha1')
    let checksum = shasum.update(JSON.stringify(body)).digest('hex')
    if (checksum !== decoded.checksum) {
        console.log('Checksum Mismatch')
        return false
    }

    return true
}

app.post("/", (req, res) => {
    // Extract necessary request information.
    let body  = req.body
    let token = req.get('X-Dt-signature')
    
    // Validate request origin and content integrity.
    if (verifyRequest(body, token) === false) {
        res.sendStatus(400)
        return
    } 

    // Log the request body.
    console.log(body)
    
    res.sendStatus(200)
})

app.listen(process.env.PORT)

In the same directory, install the following packages. This updates package.json so that Heroku can replicate the environment when deploying.

npm install body-parser==1.20.2
npm install express==4.18.2
npm install jsonwebtoken==9.0.1

Create a file Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where node index.js should be used to serve the app.

web: node index.js

Go 1.20

In the directory, initialize a go application with the following command. This generates the file go.mod which specifies version and requirements. Give any name you like.

go mod init example

Create a new file main.go where you paste the following snippet which contains the server code for receiving and validating the Data Connector request.

package main

import (
	"crypto/sha1"
	"encoding/hex"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"

	jwt "github.com/golang-jwt/jwt/v5"
)

// Environment variables.
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")

// verifyRequest validates the request origin and content integrity.
func verifyRequest(bodyBytes []byte, tokenString string) error {
	// Decode the token using signature secret.
	claims := jwt.MapClaims{}
	_, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
		return []byte(signatureSecret), nil
	})
	if err != nil {
		return err
	}

	// Verify the request body checksum.
	sha1Bytes := sha1.Sum(bodyBytes)
	sha1String := hex.EncodeToString(sha1Bytes[:])
	if sha1String != claims["checksum"] {
		return fmt.Errorf("Checksum mismatch.")
	}

	return nil
}

func endpoint(w http.ResponseWriter, r *http.Request) {
	if r.URL.Path != "/" {
		http.Error(w, "404 not found.", http.StatusNotFound)
		return
	}

	// Take an action depending on the request method.
	switch r.Method {
	case "POST":
		if err := r.ParseForm(); err != nil {
			fmt.Fprintf(w, "ParseForm() err: %v", err)
			return
		}

		// Attempt to extract the body bytes of the request.
		body, err := ioutil.ReadAll(r.Body)
		if err != nil {
			log.Printf("Error reading body: %v", err)
			http.Error(w, "Can't read body", http.StatusBadRequest)
			return
		}

		// Validate the origin and contents of the request.
		if err = verifyRequest(body, r.Header.Get("X-Dt-Signature")); err != nil {
			log.Printf("Could not validate request: %v", err)
			http.Error(w, "Could not validate request", http.StatusBadRequest)
			return
		}

		//
		// Further processing here.
		//

		log.Println("Success")

	default:
		http.Error(w, "Only POST methods are allowed.", http.StatusMethodNotAllowed)
	}
}

func main() {
	http.HandleFunc("/", endpoint)

	port := ":" + string(os.Getenv("PORT"))
	log.Printf("Listening on port %s.", port)
	if err := http.ListenAndServe(port, nil); err != nil {
		log.Fatal(err)
	}
}

In the same directory, run the following command to install the required package.

go get github.com/golang-jwt/jwt/[email protected]

Build your application into a new directory bin.

GOOS=linux GOARCH=amd64 go build -o bin/main -v .

Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where the app should be server from the binary bin/main that we built in the previous step.

Procfile

web: bin/main

Deploy the App

The following steps will deploy your application to a new app in Heroku. Take note of the app name given to the application in step 4.

1. Initialize your app's directory as a Git repository: git init

2. Add all changes: git add .

3. Commit the changes: git commit -m "initial commit"

4. Create the Heroku app, and add it as a Git remote: heroku create <APP_NAME>. You can pick a unique app name yourself or let Heroku create a random one by omitting <APP_NAME>. This will be used as a part of the URL for your server.

5. Push your application to Heroku: git push heroku main

Post Deployment Configuration

Add a new signature secret configuration variable to your app using the following command. This can be read by the python instance and imported as an environment variable to use. Use a strong secret.

heroku config:set -a <APP_NAME> DT_SIGNATURE_SECRET=<YOUR_SECRET>

Your app is now ready to receive requests, but we need to know the endpoint URL. This can be acquired by looking at the Web URL field in the results of the following command. Save it for later.

heroku apps:info -a <APP_NAME>

Create a Data Connector

To continuously forward the data to our newly created Heroku server, a Data Connector with almost all default settings is sufficient. Refer to Creating a Data Connector if you are unfamiliar with the process. The following configurations should be set.

• Endpoint URL: The Web URL found in the previous step.

• Signature Secret: The value of DT_SIGNATURE_SECRET environment variable.

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.

Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each Periodic Heartbeat or by touching a sensor to force a new event.

If instead the Error counter increments, a response containing a non-2xx status code is returned.

• Verify that the Data Connector endpoint URL is correct.

• You can view the logs of your Heroku app with the following command.

  heroku logs --app=<APP_NAME> --tail