# Performance monitoring using Node.js, socket.io and MarkLogic
Source: https://tpiros.dev/blog/performance-monitoring-using-node-js-socket-io-and-marklogic
I've worked a lot with sockets before, and recently I needed a good example application to demonstrate them beyond a chat app. The idea I landed on: gather metrics from Node.js and plot the values on a chart. Then I pushed it further. Wouldn't it be nice to save that data in a database for historical reporting later? A few hours of work and the app was born.
> The source code for the app can be found here: [https://github.com/tpiros/system-information](https://github.com/tpiros/system-information)
> Please note that this application has been written using ES2015
## Requirements
The application should gather data using the built-in 'os' library from Node.js and display it on a chart. For charting I went with [Google Charts](https://developers.google.com/chart/) because it's quick to set up. The other requirement: save the collected data to a MarkLogic database.
The logical starting point is the Node.js application itself.
## Setting up [socket.io](http://socket.io)
When I'm building a Node.js app that needs a browser view, I reach for Express. If you haven't worked with Express before, check its [documentation](http://expressjs.com/en/4x/api.html). The boilerplate Express app with [socket.io](http://socket.io) wired in looks something like this:
```js
'use strict';
const express = require('express');
const app = express();
const server = require('http').Server(app);
const io = require('socket.io')(server);
const os = require('os');
const socket = require('./socket').connect(io, os);
const hbs = require('express-handlebars');
const router = express.Router();
app.set('port', 8080);
app.use('/', router);
app.use('/bower_components', express.static(__dirname + '/bower_components'));
app.use('/js', express.static(__dirname + '/js'));
app.engine('.hbs', hbs());
app.set('view engine', '.hbs');
let indexRoute = (req, res) => {
res.render(__dirname + '/index');
};
router.route('/').get(indexRoute);
server.listen(app.get('port'), () => {
console.log('Magic happens on port ' + app.get('port'));
});
```
Those few lines grab all the dependencies, set the view engine to [handlebars](http://handlebarsjs.com/), render the `index.hbs` file, and start the HTTP server on port 8080.
Notice that line 7 includes a file called `socket.js` and calls a `connect()` method with two parameters, `io` and `os`.
### Separate socket.js
I like keeping application logic clean and separated, so I created a dedicated file for all [socket.io](http://socket.io) logic. `socket.js` is where the actual socket connection lives.
The logic is simple. A connection is made to the socket, then the [Node.js `os`](https://nodejs.org/dist/latest-v4.x/docs/api/os.html) library collects system information. Every 5 seconds, a message containing the collected metrics gets emitted via `setInterval()` and `socket.emit()`:
```js
const os = require('os');
var connect = (io, os) => {
io.on('connection', (socket) => {
var load = os.loadavg()[0];
var totalMemory = os.totalmem();
var freeMemory = os.freemem();
var usedMemory = Number((totalMemory - freeMemory) / 1073741824).toFixed(4);
socket.emit('resources', { cpu: load, memory: usedMemory });
setInterval(() => {
load = os.loadavg()[0];
freeMemory = os.freemem();
usedMemory = Number((totalMemory - freeMemory) / 1073741824).toFixed(4);
socket.emit('resources', { cpu: load, memory: usedMemory });
}, 5000);
});
};
module.exports.connect = connect;
```
You might wonder why there are two `socket.emit()` calls. The first one (line 9) fires when the application starts up. The second one (line 14) fires 5 seconds later, and every 5 seconds after that. `socket.emit()` pushes data out, and clients connecting to this socket can listen for the `resources` message to grab it.
The data sent to the template has a simple JSON structure:
```js
{
cpu: 'some value',
memory: 'some other value'
}
```
That wraps the backend. Let's look at how this data gets displayed in the browser.
## Creating the client
Here's a simple `index.hbs` file that includes the Google Charting API and the socket connection:
```html
System Information
```
`charting.js` collects the data for the chart and draws it. The code is fairly short. If you want to tweak the chart, take a look at the [Google Charting API documentation](https://developers.google.com/chart/interactive/docs/).
```html
(function() { 'use strict'; google.charts.load('current',
{'packages':['corechart']}); google.charts.setOnLoadCallback(drawChart); var
socket = io.connect('http://localhost:8080'); function drawChart() { var options
= { title: 'System Utilisation', curveType: 'function', legend: { position:
'bottom' }, pointSize: 3 }; var chart = new
google.visualization.LineChart(document.getElementById('curve_chart')); var
dataArray = [['Time', 'CPU Average (%)', 'Used Memory (GB)'], [new Date(), 0,
0]]; var data = google.visualization.arrayToDataTable( dataArray );
chart.draw(data, options); socket.on('resources', function (load) {
dataArray.push([new Date(), load.cpu, load.memory]); data =
google.visualization.arrayToDataTable( dataArray ); chart.draw(data, options);
}); } })();
```
Line 5 connects to the socket created earlier (the HTTP server runs on port 8080, so the connection points to `http://localhost:8080`).
Lines 21 to 27 are where the action happens. Remember how the Node.js app uses `socket.emit()` to emit the `resources` message with data? On the client, `socket.on()` listens for that message, and the `load` argument in the callback contains the server's data. The data slots into an array, the chart draws, and it redraws every time the server emits a new `resources` message.
That covers the basics. At this point a chart appears in the browser showing data collected via the Node.js `os` library.
## Persisting data in MarkLogic
Why stop there? The application can do more by displaying extra system information and saving the utilisation metrics to a database for later retrieval.
Back in the Node.js code, let's collect some additional metrics and feed them to the handlebars template:
```js
let dataObject = {
osType: os.type().toLowerCase() === 'darwin' ? 'Mac OS X' : os.type(),
osReleaseVersion: os.release(),
osArch: os.arch(),
osCPUs: os.cpus(),
osHostname: os.hostname(),
osTotalMemory: Number(os.totalmem() / 1073741824).toFixed(0),
};
// etc
let indexRoute = (req, res) => {
res.render(__dirname + '/index', { data: dataObject });
};
```
Let's also persist this data by adding a few lines to the Node.js application:
```js
db.documents
.write({
uri: '/data/host.json',
contentType: 'application/json',
content: dataObject,
})
.result()
.then((response) => {
console.log(response.documents[0].uri + ' inserted to the database.');
})
.catch((error) => {
console.log(error);
});
```
> If you'd like to know how to connect to the MarkLogic database and learn more about the Node.js Client API please read this article.
The CPU and memory metrics should be persisted too, but the question is how. Ideally, one document gets created per data collection. That means modifying `socket.js` (where the data collection happens). The new `setInterval()` looks like this:
```js
setInterval(() => {
load = os.loadavg()[0];
freeMemory = os.freemem();
usedMemory = Number((totalMemory - freeMemory) / 1073741824).toFixed(4);
socket.emit('resources', { cpu: load, memory: usedMemory });
db.documents
.write({
uri: '/data/' + Date.now() + '.json',
contentType: 'application/json',
content: { cpu: load, memory: usedMemory },
})
.result()
.then((response) => {
console.log(response.documents[0].uri + ' inserted to the database');
})
.catch((error) => {
console.log(error);
});
}, 5000);
```
Documents are stored with the `URI` (unique document identifier) of `/data/EPOCH.json`. (If you're not sure how URIs work in MarkLogic, refer to the article I linked earlier.)
All that's left is displaying the data in the handlebars template:
```html
System CPU: {{ data.osCPUs.0.model }},
Total memory: {{ data.osTotalMemory }}GB
```
## Conclusion
The application is now complete. It collects CPU and memory utilisation every 5 seconds and saves that data to the MarkLogic database with the same structure discussed earlier.
In a later article we'll look at how to run 'historical' reporting on the collected data using documents in the database. Stay tuned!