System Integration using Emerging Technologies

System Integration using Emerging Technologies Graham Oakes

Infrastructure Solution Architect, Messaging Lead

Shruti Appiah

Systems Design Engineering Student, University of Waterloo

MQ Light, Node-RED and Docker Integration

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

Introduction

• The MQ Micro Server (legacy MQ application) • High level and why we choose to use them • MQ Light • Node-RED • Docker

• Other Components

• Oracle Virtual Box • Ubuntu Desktop (Linux)

Message Flow Diagram

Agenda

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

Message Flow Diagram

Web Browser Web Browser

Input Accounts “1,2,4”

IN

HTML Form

Node-RED

HTTP IN (Form)

Message Flow Diagram

Node-RED: Preparation Flow OUT - JSON

IN - HTML Form

HTTP IN (Form) Receives HTTP Input

Pull out Accounts Pulls accounts, creates JSON

Websphere MQ & MQLight MQLight SUB Accounts Req

Node-RED MQLight PUB Accounts Req Publish on MQ Light PUB

MQLight Published JSON Request Message

Message Flow Diagram

MQ, MQ Light, and MQ Micro Service MQLight Published JSON Request Message MQLight SUB Accounts Req

MQ Queue

MQ Sub convert to MQ Queue

MQ Micro Service

Accounts Enrichment

MQLight PUB Accounts Resp

MQMS: Enrich and MQ Light PUB MQLight Published JSON Response Message

Message Flow Diagram

Node-RED: Log MQLight Published JSON Response Message MQLight SUB Accounts Response

Log Response

MQ Light SUB

Logger

Message Flow Diagram

Messaging Workflow Web Browser

IN HTML Form

OUT - JSON

HTTP IN (Form)

Pull out Accounts

MQLight Published JSON Request Message MQLight SUB Accounts Req

* Node-RED

* Websphere MQ & MQLight

* MQ Micro Service

Accounts Enrichment

MQ Queue

MQLight PUB Accounts Req

MQLight PUB Accounts Resp

MQLight Published JSON Response Message MQLight SUB Accounts Resp

Log Response

* Node-RED

Message Flow Diagram

Infrastructure

LAPTOP

 Virtual Box – 2 images • • • •

MQ and MQ Light Node-RED Network: Host-only adapter Ubuntu Desktop 14.04

 Docker running on same server as Node-RED

HOST-ONLY ADAPTER

 All components on one laptop

Virtual Box

Websphere MQ & MQLight

Node-RED

Virtual Box

Docker MQ Micro Service

Message Flow Diagram

Message Flow on Infrastructure Static Webpage on Laptop

T

MQ Sub

REQUEST

LAPTOP

Q

MQ Queue

QUEUE

T

MQ Sub

RESPONSE LOG

HTTP IN ENRICHMENT

Virtual Box

Websphere MQ & MQLight

Virtual Box Node-RED

Docker MQ Micro Service

Agenda MQ Micro Service

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

MQ Light MQ Light

• Provides a easy to install, use and configure pub/sub engine • URL: http://www.ibm.com/developerworks/downloads/ws/wmq/ • Installation modes • Standalone or • In with WebSphere MQ 8.0.0.2 FP2

• Standalone • • • •

On Ubuntu – simple, any directory, any user, gunzip and tar –x Start: mqlight-start (2 mins if you’re a slow typer) Apps connect to port 5672 Admin, AD and DOCs on: http://localhost:9180/#page=docs

MQ Light

Install MQ Light on MQ 8.0.0.2 fp2 • • • • • •

RPM Installs Under /opt/mqm/amqp amqplap.sh # Accept the license strmqm -e CMDLEVEL=801 $QM # Enables latest functions, restart ALTER service(SYSTEM.AMQP.SERVICE) control(QMGR) ALTER CHANNEL(SYSTEM.DEF.AMQP) CHLTYPE(AMQP) MCAUSER('mqm') START channel(SYSTEM.DEF.AMQP)

• DEFINE LISTENER(LISTENER.TCP) TRPTYPE(TCP) CONTROL(QMGR) PORT(1414)

• Restart QMGR

Verification

MQ Light

• dis svstatus(SYSTEM.AMQP.SERVICE) SERVICE(SYSTEM.AMQP.SERVICE) STATUS(RUNNING) PID(3042) SERVTYPE(SERVER) CONTROL(QMGR) STARTCMD(/opt/mqm//bin/amqp.sh) • dis CHSTATUS(SYSTEM.DEF.AMQP) CHLTYPE(AMQP) CHANNEL(SYSTEM.DEF.AMQP) CHLTYPE(AMQP) CONNECTIONS(0) STATUS(RUNNING)

• $ ps -ef | grep 3042 # 3042 from output above mqm 3042 3017 0 17:24 ? 00:00:00 /bin/sh /opt/mqm//bin/amqp.sh start -m … • $ netstat -an | grep 5672 tcp6 0 0 :::5672

DEMO

:::*

show commands above

LISTEN

Agenda MQ Micro Service

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

MQ Micro Service

Q

ACCOUNTS.REQUESTS

JMS Queue Receive

New Msg: Accounts Enrichment

JMS Topic Publish

Accounts/ Response

T

Accounts/Response

MQ Micro Service

 A simple legacy JMS 1.1 Application  Listens on a JMS destination “ACCOUNTS.REQUESTS”.  When JSON message arrives, parses it and creates a new enriched message { "accounts":["1","2","4"] }

INPUT

{ "accountsResponse": [ { "acc_id": "1", "name": "Jenny Chow", "sex": "f", "membersince": "1996-01-21" }, { "acc_id": "2", "name": "Shruti Appiah", "sex": "f", "membersince": "2015-01-31" }, { "acc_id": "4", "name": "Emily Peyton", "sex": "f", "membersince": "2005-10-31" }] }

 Publish messages to “Accounts/Response”  Developed in Java with IDE Eclipse. Deployable to JRE

OUTPUT

MQ Micro Service – MQ Configuration

MQ Micro Service

Simplifier the security – don’t do this on a share environment • ALTER QMGR CHLAUTH(disabled) CONNAUTH('')

Define Application MQ components • DEFINE channel(ACCOUNTS.SVRCONN) chltype(svrconn) mcauser('mqm') SSLCAUTH(OPTIONAL) • define ql(ACCOUNTS.REQUESTS) • define topic('Accounts/Request') topicstr('Accounts/Request') • define sub('Accounts/Request') TOPICOBJ('Accounts/Request') DEST(ACCOUNTS.REQUESTS) • define topic('Accounts/Response') topicstr('Accounts/Response')

MQ Micro Service – JNDI Configuration

MQ Micro Service

• Define these objects in MQExplorer • We used a file context (other options are LDAP, WAS) • JNDI Objects required • JMS Connection Factory • JMS Destination: “InputQ” • JMS Destination: “Output2”

 Queue Mgr  Q: ACCOUNTS.REQUESTS  T: “Accounts/Response”

• Stored in a file called “.bindings” which can be moved around • We placed under “/var/mqm/jmsobjects” cd /var/mqm/jmsobjects ls -la -rw-rw-r-- 1 mqm mqm 31784 Aug 30 19:41 .bindings

MQ Micro Service - Running MQ Micro Service

• Files (files packaged in as.jar) • AccountsService.java • MessageProcessing.java • MQ and JSON Jars in a folder

The JMS application, jar entry point The JSON parser and enrichment Java Libraries

• AccountsService.properties Holds properties, such as jndi obects • .bindings Holds all the JNDI objects • Create a directory on Node-RED server and copy files across • Install Java Run-Time • java -jar as.jar

MQ Micro Service - Running

MQ Micro Service

• Start the Micro Server java -jar as.jar

Propery File [AccountsService.properties] initialContextUrl [file:.] contextFactory [com.sun.jndi.fscontext.RefFSContextFactory] connectionFactoryFromJndi [QM1] inputDestinationFromJndi [inputQ] outputDestinationFromJndi [output2] timeout [50000] Initial context found! cf[QM1] Destinations: Input[queue:///ACCOUNTS.REQUESTS], Output[topic://Accounts/Response] Connection, session and Consumer – created connection - started Wait 50000ms for a message

DEMO – Start MQ MS, put msg

• Put a message onto the queue

echo '{ "accounts":["1","2","4"] }' | /opt/mqm/samp/bin/amqsput ACCOUNTS.REQUESTS QM1

Received message:{ "accounts":["1","2","4"] } InMessage[{ "accounts":["1","2","4"] }] acct_id[1] acct_id[2] acct_id[4] Sent message:{ "accountsResponse": [{ "acc_id": "1", "name": "Jenny Chow", "sex": "f", "membersince": "1996-01-21" }, { "acc_id": "2", "name": "Shruti Appiah", "sex": "f", "membersince": "2015-0131" }, { "acc_id": "4", "name": "Emily Peyton", "sex": "f", "membersince": "2005-10-31" }] } Wait 50000ms for a message

Agenda

Node-RED

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

Node-RED

Node-RED

• Home page: http://nodered.org/ • Tool for wiring together hardware devices, APIs and online services in new and interesting ways. • browser-based flow editor that makes it easy to wire together flows using the wide range nodes in the palette. • Deployed to the runtime in a single-click. • JavaScript functions can be created within the editor using the rich text editor. • A built-in library allows you to save useful functions, templates or flows for re-use.

Node-RED – Installation (Ubuntu)

Installed pre-req software • sudo apt-get install nodejs • sudo apt-get install npm • sudo apt-get install g++

# Node JS # Node Package Manager # g++

Node-RED

Install Node-RED and MQ Light nodes • sudo npm install -g node-red • sudo npm install -g node-red-node-mqlight Start it up! • node-red • Listens on port 1880

Message Flow

Node-RED

MQLight Nodes

MQLight OUT node

Provides an MQLight send client. Node-RED

Sends the value passed in to the specified Topic

• Similarly, the MQLight IN node subscribes the client to a destination based on the supplied topic.

Node-RED - Flows

Node-RED

HTTP IN and Test Flow • http in: port 1880 / context “in”. • Convert to java script object • Create a list of accounts • Convert to JSON for transmission • Finally publish to MQ Light on topic: Accounts/Request and send the same message back to the browser. • Side note: Test drive the flow: save time when developing. Display the response message • Listens on a MQ Light topic: Accounts/Response (side note: which is published by the MQ Micro Service) • Logs the response message

DEMO

Show flows, deployment, test drive flow

Running Node-RED on node.js

After installing the Node-RED npm package and required modules, enter node-red to start the application.

Node-RED

Agenda

Static Web Page

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

Simple Static Web Page

Static Web Page

• Used to kick off the message flow • Calls Node-RED’s HTTP In node • Port 1880, context “in” • Passing in a form which holds the account list 0,1,2

Simple Static Web Page

Static Web Page

Enter your account details here

Account numbers:

DEMO

Agenda

Docker

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

Docker

Docker

• Docker is a Virtual Environment not a Virtual Machine • It segregates, using name spaces, all the system resources • A Docker image is normally relatively small • It runs in Linux on Linux Containers • Our Virtual Box image running Node-RED will run Docker • We can have many Docker images running at the same time • This is perfect for small servers or micro services – MQ Micro Service… • URL: https://www.docker.com/

Docker - Installation

Install • sudo apt-get update • wget -qO- https://get.docker.com/ | sh

# Install Docker

Add user to docker group • sudo usermod -aG docker graham • Note: The docker group is like root on an OS so very powerful! Docker

Start the Linux Docker service • sudo service docker start

Docker - Commands

• docker images • docker search fedora • docker pull fedora • docker images • docker run -it fedora /bin/bash

Docker

DEMO

# Shows local images # Search in the external repo # Pull fedora to our local repo # Will now include fedora # Start the bash shell into fedora,

list images, run image, run 2 images

Docker - File

• A Docker file allows you to script the creation of an image. • We used it for the MQ Micro Service • We don’t need to use the same Linux distribution • Lets use Fedora as we already have that

Docker

Dockerfile for MQ Micro Service

Docker

# Docker file to create MQ Micro Service FROM fedora MAINTAINER Graham Oakes LABEL Description="This image is used to start the AccountsService" Vendor="GOSA Products" Version="1.0" # Copy the java code over RUN mkdir -p /apps/AccountsService ADD . /apps/AccountsService/ ADD as_lib/ /apps/AccountsService/ RUN rpm -Uvh /apps/AccountsService/jre-7u79-linux-x64.rpm CMD cd /apps/AccountsService; java -jar as.jar $

DEMO create image, run it

Agenda

All Together

• Message Flow Diagram • MQ Light • MQ Micro Service • Node-RED • A Simple Static Web Page • Docker • Pull it all together

Putting It All together Initial Status • Node-RED • Down

• MQ Server • Up

• Docker MQ Micro Server All Together

• Down

Laptop 192.168.56.2 T

Request

Q

REQUESTS

T

Response

Log

HTTP In

Enrichment

Virtual Box WebSphere MQ and MQ Light 192.168.56.102

Docker MQ Micro Service

Virtual Box Node-Red 192.168.56.103

Putting It All together

1. Start index.html in a browser, enter “0,1” and hit submit. Expectation: This webpage is not available - as Node-RED is down 2. Startup Node-RED and perform the same tasks Expectation: JSON on browser and a message sitting on the input queue for the application echo "dis ql(ACCOUNTS.REQUESTS) curdepth" | runmqsc QM1

All Together

3. Startup Docker MQ Micro Service. Expectation: Message from 2 to get processed

#1

• See information pop up in the Docker window as the message is processed • See information in the Node-RED window as the publication is returned

Putting It All together

4. Perform same task. Expectation:

• JSON on browser • See information pop up in the Docker window as the message is processed • See information in the Node-RED window as the publication is returned

5. Startup a second Docker MQ Micro Service, and perform same task many times. Expectation: As before however the Docker applications will process them most likely in a round robin fashion. All Together

NOTE: We have submitted a paper to DeveloperWorks to show others how to build this demo

Questions 1

• Which images are typically bigger Docker or Virtual Box?

Question 2

• We installed MQ Light with MQ. • What other way can MQ Light be deployed?