I have a microcontroller and I want to upload sensor data to the internet. I also want a database to store sensor readings and a nice gui so that others can use it. Setting up my own server and maintaining it is not worth the effort. So I want a cloud service, more precisely a free cloud service.

This memo is a write-up of one way to set up the required software infrastructure. I will describe hardware components like sensors in another memo.

The project files can be cloned from the compagnion repo.

1. Introduction

IBM Bluemix offers a cloud service called IBM Cloud or IBM Bluemix.[1] Several plans are available, I will use the Lite Plan also known as the Free Plan.

Bluemix provides Cloud Foundry, a Platform-as-a-Service infrastructure that will allow us to run our application and a database without any installation chores. Bluemix also provides the IBM Watson IoT Platform, hereafter referred to as IoT Platform. This is the central communication hub for MQTT messaging. Devices can publish messages and subscribe to commands emitted by application.

Node-RED is a programming environment specifically designed for IoT requirements.

I first created an account on IBM Bluemix (with the Lite/Free Plan) and subsequently went through the following steps:

  1. understand the required terminology

  2. survive credentials hell

  3. set up a git repo where your code will reside

  4. develop and test the application locally

  5. deploy the application to bluemix

  6. test my work on the desktop (rather than in the cloud).

The application, also known as the app hereafter, is based on this IBM developer tutorial albeit in a simplified form. The motivation for adopting the work flow discussed in this tutorial is derived from two posts by Nick O’Leary:

The main difference with respect to the first link is that Bluemix now provides an integrated Continuous Delivery Toolchain, which I use in my setup.

2. Terminology

As far as this memo is concerned, Bluemix offers Cloud Foundry Services, Services and allows to develop Cloud Foundry Apps.

This can be seen on the Dashboard:

Dashboard
Figure 1. The Bluemix Dashboard

More important for this memo is the Resource list (in the menu):

Resource list
Figure 2. The Bluemix Resource list

These are the items to create:

  • Cloud Foundry Apps: develop the My-Node-Red-App, a Node-RED flow-based javascript application without writing a single line of code.

  • Cloud Foundry Services: create the My-IoT-Platform service that acts as a MQTT broker, amongst other tasks. In particular, configure MQTT communications between my devices and the database.

  • Cloud Foundry Services: create the My-Node-RED-App-cloudantNoSQLDB database, more precisely an alias to the underlying database service. This alias is created automatically.

  • Services: Bluemix creates Cloudant-ay as the underlying database service where I store sensor readings. The name is choosen by Bluemix during setup. Your name will be different.

  • Services: Continuous Delivery: the toolchain service enabled for the app. It includes a git repo and a delivery pipeline that deploys the app when pushed to the repo.

  • Developer Tools: My-Node-RED-App represents the actual toolchain associated with the app.

2.1. Dashboards

In addition to the initial Bluemix Dashboard ( which I found not useful for this memo), there is the IoT Platform dashboard and the Cloudant database dashboard:

IoT Platform Dashboard
Figure 3. The IoT Platform Dashboard

The IoT Platform Dashboard is used to create Device Types and Devices.

Cloudant Dashboard
Figure 4. The Cloudant Dashboard

The Cloudant Dashboard will contain sensor data. The node-red database is created by Bluemix during setup.

2.2. Credential hell

One of the most intimidating aspects when using the cloud for the first time seems to be the avalanche of credentials, passwords, access tokens, ids, and other access codes.

Access code is a generic name I choose to refer to any kind of 'secret' that is required to use Bluemix and the services it provides. Here is a preview that should help to mitigate that onslaught.

Table 1. Access codes
Access Code Name Context provided by Why ? Example

OrgId

OrgId

IoT Platform Dashboard → top right corner

Bluemix

required for MQTT communications

nhr9i3 (6 char string)

Bluemix

IBM Cloud Api Key

Dashboard → Manage → Access IAM → IBM Cloud Api Keys

Bluemix

required by the Continuous Delivery Toolchain service

pmt_DmpY_nL6ExU8FhAk5C4f4-l73no <trunc>

IoT Platform

Api Key

IoT Platform Dashboard → Apps → Generate API Key

Bluemix

mqtt clients need these credentials to access the platform

API Key: a-m60i9s-gjv4sdf2t Auth Token: fiL1QgUEknggdfg88

DAT

Device Authentification Token

IoT Platform Dashboard → Devices → Add Device → …​. → Device Authentification Token

User

mqtt devices need this token to access the platform

any string

git

SSH key

Toolchain → git → top right corner → Settings → SSH Keys

ssh-keygen

to clone and push the project’s repository

ssh-rsa AAAAB3NzaC1yc2EA <trunc>

VCAP

VCAP_Services enironment variable

My-Node-RED-App → Runtime → Environment variables → Export

Bluemix

to be able to locally access remote services like the Cloudant database

{ "cloudantNoSQLDB": [ <trunc>

NR-flow

Node-RED credential secret

Project specific files

User

to decrypt flow.json, the file that defines a Node-RED application

any string

NR-editor

Node-RED editor access

Project specific files

User

to access the editor and possibly change the flow.json file

username: admin, password: $2a$08$LUhxY <trunc>
After creating an access code on Bluemix, it must be copied and saved immediately. Some codes can’t be retrieved at a later stage. Also, these codes don’t belong in the project’s repository.
In the following, I assume orgId = nhr9i3, your orgId will be different.

Create the Bluemix access code:

Dashboard → Manage → Access IAM → IBM Cloud Api Keys → Create an IBM Cloud API key

3. Create the My-Node-RED-App and the associated infrastructure

Create three items:

  1. the application itself,

  2. the database used to store sensor data.

  3. the build and deploy delivery toolchain

there are several Node-RED-Bluemix tutorials on the internet where the database also stores the Node-Red flow files. This is suitable for a work flow where development is done remotely. However, I prefer a work flow where all development is done locally and the flow files are then pushed to repository from where they are deployed automatically.

3.1. Create the application and the database

  1. create a Bluemix access code if not already done.

  2. from the Bluemix Dashboard page, click on the Catalog menu.

  3. from the Starter Kits Category, click on Node-Red Starter

  4. enter the App name: My-Node-RED-App

  5. selected Plan: Lite

  6. select the Cloudant service to be connected to the app:

Cloudant
Figure 5. The Cloudant Service link

This opens the Getting started pane of the application’s dashboard:

App Dashboard
Figure 6. The My-Node-RED-App Dashboard

Ready to activate Toolchain support.

3.2. Create the continuous delivery toolchain

  1. Click Overview

  2. on the bottom right Continuous delivery click on Enable.

  3. in the Repository Type drop down menu, select New.

  4. enter the Bluemix access code

  5. click on Create

This is the dashboard of the Continuous Delivery Toolchain:

Toolchain
Figure 7. The Continuous Delivery Toolchain

Get access to this repository:

  1. click on the Git tile.

  2. set up the git access code using a public SSH key: top right corner icon → Settings → SSH Keys

  3. copy the link to the repository and clone it:

Git
Figure 8. The new empty repository for the app
$ git clone git@eu-gb.git.cloud.ibm.com:<your name>/My-Node-RED-App.git
the directory My-Node-RED-App into which I cloned the repository is called the project directory.

Go back to the application’s dashboard, then click on Runtime → Environment variables → VCAP_SERVICES → Export to export this environment variable to a file named My-Node-RED-App_vcap.json into the project directory.

This file now contains all the Bluemix specific data needed to run the app locally.

4. Create the My-IoT-Platform service

I like the following video tutorials prepared by IBM:

and the following documents:

On the Bluemix Dashboard, click on Catalog → Internet of Things → Internet of Things Platform to create an instance of the IoT Platform service. Choose My-IoT-Platform as name. Once the instance has been created, I get another dashboard:

Git
Figure 9. The dashboard of the My-IoT-Platform service instance

4.1. Generate the IOT Platform access code:

  1. click on the Launch button.

  2. on the IoT Platform Dashboard click on Apps, and on the top right corner click on Generate Api Key.

  3. leave the defaults and click Generate Key.

  4. copy the API Key and the Authentification Token right now because you won’t be able to see the again.

Remember: I refer to this pair of strings (API Key,Authentification Token) as the IoT Platform access code, see Access codes. I need it later to modify the Node-RED flow application template file. This allows me to access the remote Cloudant database when developing the app locally.

4.2. Configure connection security

To simplify testing, I change the default setting for connection security:

  1. in the IoT Dashboard, click on Security → Connection Security and edit the Default Rule.

  2. in the Default drop down menu, select TLS Optional.

Note that this is not recommended for a production environment. The reason for this change is that the mqtt node.js module I use as test mqtt client doesn’t seem to use ssl out of the box and I wanted to keep things as simple as possible.

See also Configuring security policies.

4.3. Configure Devices

Device type creation and device creation is explained in the tutorial material mentioned at the beginning of this chapter and therefore the following tasks are simple:

  1. create a new DeviceType with name nodejs-client

  2. and a device of this type with name nodejs-client-1

  3. click until your are asked to provide a Device Authentication Token.

I refer to this token as DAT access code, see Access codes. I need it later to instantiate the mqtt client test program template.
write down the access code, it can’t be retrieved once the page is closed.

The dashboard should now look like this:

IoT Platform Dashboard
Figure 10. The IoT Platform Dashboard after device creation

Ready to add and modify the application project files.

5. Install local software

Install the Node-RED stack and the IBM Cloud (CLI) on Linux Mint 18.

5.1. Node-RED

There are many ways to install Node-RED. I choose the following:

$ curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
$ sudo apt-get install -y nodejs
$ sudo npm install -g node-red (1)
$ sudo npm install -g mqtt (2)
1 This might result in:
gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/lib/node_modules/node-red/node_modules/bcrypt/.node-gyp'
I used the --unsafe-perm option to fix the problem:
$ sudo npm install -g --unsafe-perm node-red
2 since the test client uses the require mechanism and I installed the mqtt module globally, NODE_PATH must include the global node_modules directory tree, for example:
export NODE_PATH=/usr/lib/node_modules

The later node module is required for the mqtt test client.

I also need node-red-admin to generate a password hash:

$ sudo npm install -g node-red-admin

5.2. Bluemix CLI

The IBM Cloud Command Line Interface (CLI) is a tool to monitor all aspects of the remote infrastructure from the local terminal. The most important function for this memo is the visualization of log files.

Install the IBM Cloud CLI [https://github.com/IBM-Cloud/ibm-cloud-cli-release/releases].

6. Instantiate template files

The My-Node-RED-App consists of two sets of files:

  • files that are project agnostic and can be used without modification,

  • templates files that needs to be populated with project specific data, mostly access codes.

These files are available in the compagnion repo.

6.1. Project agnostic files

These include:

  • .bluemix/pipeline.yml: used by the Delivery pipeline of the Continuous Integration Toolchain.

  • flow.json: the Node-RED application

  • package.json: required by node.js

  • run-app.sh and run-client.sh: shortcuts to run the app and the mqtt test client locally.

6.2. Project specific files

These files are templates and require customisation:

Table 2. template files
Name Function Changes

.gitignore

exclude files from being committed

app name → My-Node-RED-App since the file with VCAP access code My-Node-RED-App_vcap.json is required only locally, and should not be exposed.

manifest.yml

tells the remote deploy stage how to build and start the app

app name → My-Node-RED-App

settings.js

the Node-RED settings file, see the Node-RED documentation

access codes: NR-flow, NR-editor, VCAP, see Access codes

mqtt-client.js

used to publish messages and test the app

access codes: orgId, DAT and clientId

The password property in the settings.js file for the NR-editor access code should be a hash rather than a plain text string.

Generate the hash:

$ node-red-admin hash-pw
Password: <enter your NR-editor password>
$2a$08$ufCvyA.kLcV4LRdCTLFsruX.ejTq1UkxiI4FLt3oLpXvs/J1dlGA.

The result is the hash of the NR-editor password that goes into settings.js.

Here are some hightlights of the required changes in the My-Node-RED-App project directory:

manifest.yml
  applications:
    - path: .
      name: My-Node-RED-App (1)
      environment_json: {}
      memory: 256M
      instances: 1
      disk_quota: 1024M
      services:
        - My-Node-RED-App-cloudantNoSQLDB (2)
      command:  node-red -s settings.js -u .
1 application name
2 name of the connection between the app and the database.
settings.js
// ...
var vcap = "My-Node-RED-App_vcap.json"; (1)
// ...
credentialSecret: "My-Node-RED-AppCredential", (2)
adminAuth: {
  type: "credentials",
  users: [{
    username: "admin", (3)
    password: "$2a$08$LUhdfsdafgasggdfgdfgerferg", (4)
    permissions: "*"
  }],
},
1 The file name containing the VCAP access code environment variable.
2 The NR-flow accesscode. I choose My-Node-RED-AppCredential. For production code replace this with an environment variable.
3 The NR-editor username access code. I choose admin.
4 The NR-editor password access code generated in the previous step.
If the NR-flow access-code is changed after the first app execution, there is - to my knowledge - no other way to apply the change than deleting the ./node_module director inside the project directory.
mqtt-client.js
// ...
var client = mqtt.connect('mqtt://nhr9i3.messaging.internetofthings.ibmcloud.com', { (1)
  "username": "use-token-auth",
  "password": "nodejs-client-1-token", (2)
  "clientId": "d:nhr9i3:nodejs-client:nodejs-client-1" (3)

})
// ...
1 assuming orgId is nhr9i3, yours will be different
2 the DAT access code
3 client identifier: d:orgId:device_type:device

Ready to finish the local installation and initialize the app.

7. Initialize the work flow

With package.json in place, finish the installation:

$ npm install

7.1. Initialize the app

The first time the app is run, Node-RED will generate a credential file for flow.json named flow_cred.json. This file contains sensitive data like the IoT Platform access code. The NR-flow access code is used by Node-RED to decrypt this file.

Run the app from the project directory:

$ ./run-app.sh

  1. enter the username provided in settings.js and the password supplied to the node-red-admin hash-pw command, see NR-editor password access code.

  2. open the IBM IoT node and enter the IoT Platform access code. Click on Deploy. This will generate the credential file flow_cred.json.

  3. populate the repo, commit and push to Bluemix:

$ git add -A && git commit
$ git push -u origin master

Open the Delivery Pipeline of the My-Node-RED-App Toolchain dashboard:

Deploy Stage
Figure 11. The Delivery Pipelines shows the build-deploy process

While the build is running, log into the IBM Cloud CLI:

$ ibmcloud login
API endpoint: https://cloud.ibm.com
Region: eu-gb (1)
Email> <enter the email address you used to register with Bluemix>
1 it is very important to set the correct region. Check the domain you selected when creating the cloud foundry app.

and set the endpoint:

$ ibmcloud target --cf

After the build has finished, check the log from with IBM Cloud CLI:

$ ibmcloud cf logs My-Node-RED-App --recent

Check the log carefully the first time the app is run. The logs might not be available immediately after the build has finished, this can take up to a minute.

Check the connection between the app and the database:

$ ibmcloud cf services
name                              service           plan                bound apps        last operation
My-IoT-Platform                   iotf-service      iotf-service-free                     create succeeded
My-Node-RED-App-cloudantNoSQLDB   cloudantNoSQLDB   Lite                My-Node-RED-App   create succeeded

Remember that My-Node-RED-App-cloudantNoSQLDB is the service name provided to manifest.yml.

8. Develop locally, Deploy remotely

Develop the application with the Node-RED editor by selecting nodes and wiring them toegether.

Run the mqtt-client:

$ ./run-client.sh
20 Aug 01:24:03 - created client
20 Aug 01:24:03 - publishing
20 Aug 01:24:04 - published: 1
20 Aug 01:24:05 - published: 2
20 Aug 01:24:06 - published: 3
^C

Go to the gui by appending ui to the url of the app.

Finally, commit and push to start the build-deploy process in the Delivery Pipeline.

Open the app’s url:

gui
Figure 12. The app running remotely

9. References

IoT tutorials

IoT development with Java

IoT with REST API

Cloudant

Node-RED & Node.js

1. I am not sure what exactly the difference is.