Performance monitoring using Node.js, socket.io and MarkLogic

I’ve worked a lot with sockets before, and recently I needed a good example application to demonstrate them beyond a chat app. The idea I landed on: gather metrics from Node.js and plot the values on a chart. Then I pushed it further. Wouldn’t it be nice to save that data in a database for historical reporting later? A few hours of work and the app was born.

The source code for the app can be found here: https://github.com/tpiros/system-information

Please note that this application has been written using ES2015

Requirements

The application should gather data using the built-in ‘os’ library from Node.js and display it on a chart. For charting I went with Google Charts because it’s quick to set up. The other requirement: save the collected data to a MarkLogic database.

The logical starting point is the Node.js application itself.

Setting up socket.io

When I’m building a Node.js app that needs a browser view, I reach for Express. If you haven’t worked with Express before, check its documentation. The boilerplate Express app with socket.io wired in looks something like this:

'use strict';
const express = require('express');
const app = express();
const server = require('http').Server(app);
const io = require('socket.io')(server);
const os = require('os');
const socket = require('./socket').connect(io, os);
const hbs = require('express-handlebars');
const router = express.Router();

app.set('port', 8080);
app.use('/', router);
app.use('/bower_components', express.static(__dirname + '/bower_components'));
app.use('/js', express.static(__dirname + '/js'));
app.engine('.hbs', hbs());
app.set('view engine', '.hbs');

let indexRoute = (req, res) => {
  res.render(__dirname + '/index');
};

router.route('/').get(indexRoute);

server.listen(app.get('port'), () => {
  console.log('Magic happens on port ' + app.get('port'));
});

Those few lines grab all the dependencies, set the view engine to handlebars, render the index.hbs file, and start the HTTP server on port 8080.

Notice that line 7 includes a file called socket.js and calls a connect() method with two parameters, io and os.

Separate socket.js

I like keeping application logic clean and separated, so I created a dedicated file for all socket.io logic. socket.js is where the actual socket connection lives.

The logic is simple. A connection is made to the socket, then the Node.js os library collects system information. Every 5 seconds, a message containing the collected metrics gets emitted via setInterval() and socket.emit():

const os = require('os');
var connect = (io, os) => {
  io.on('connection', (socket) => {
    var load = os.loadavg()[0];
    var totalMemory = os.totalmem();
    var freeMemory = os.freemem();
    var usedMemory = Number((totalMemory - freeMemory) / 1073741824).toFixed(4);

    socket.emit('resources', { cpu: load, memory: usedMemory });
    setInterval(() => {
      load = os.loadavg()[0];
      freeMemory = os.freemem();
      usedMemory = Number((totalMemory - freeMemory) / 1073741824).toFixed(4);
      socket.emit('resources', { cpu: load, memory: usedMemory });
    }, 5000);
  });
};

module.exports.connect = connect;

You might wonder why there are two socket.emit() calls. The first one (line 9) fires when the application starts up. The second one (line 14) fires 5 seconds later, and every 5 seconds after that. socket.emit() pushes data out, and clients connecting to this socket can listen for the resources message to grab it.

The data sent to the template has a simple JSON structure:

{
  cpu: 'some value',
  memory: 'some other value'
}

That wraps the backend. Let’s look at how this data gets displayed in the browser.

Creating the client

Here’s a simple index.hbs file that includes the Google Charting API and the socket connection:

<html>
  <head>
    <script src="/socket.io/socket.io.js"></script>
    <script
      type="text/javascript"
      src="https://www.gstatic.com/charts/loader.js"
    ></script>
    <script type="text/javascript" src="/js/charting.js"></script>
  </head>
  <body>
    <div class="content">
      <h1>System Information</h1>
      <div id="curve_chart" style="width: 900px; height: 500px"></div>
    </div>
  </body>
</html>

charting.js collects the data for the chart and draws it. The code is fairly short. If you want to tweak the chart, take a look at the Google Charting API documentation.

(function() { 'use strict'; google.charts.load('current',
{'packages':['corechart']}); google.charts.setOnLoadCallback(drawChart); var
socket = io.connect('http://localhost:8080'); function drawChart() { var options
= { title: 'System Utilisation', curveType: 'function', legend: { position:
'bottom' }, pointSize: 3 }; var chart = new
google.visualization.LineChart(document.getElementById('curve_chart')); var
dataArray = [['Time', 'CPU Average (%)', 'Used Memory (GB)'], [new Date(), 0,
0]]; var data = google.visualization.arrayToDataTable( dataArray );
chart.draw(data, options); socket.on('resources', function (load) {
dataArray.push([new Date(), load.cpu, load.memory]); data =
google.visualization.arrayToDataTable( dataArray ); chart.draw(data, options);
}); } })();

Line 5 connects to the socket created earlier (the HTTP server runs on port 8080, so the connection points to http://localhost:8080).

Lines 21 to 27 are where the action happens. Remember how the Node.js app uses socket.emit() to emit the resources message with data? On the client, socket.on() listens for that message, and the load argument in the callback contains the server’s data. The data slots into an array, the chart draws, and it redraws every time the server emits a new resources message.

That covers the basics. At this point a chart appears in the browser showing data collected via the Node.js os library.

Persisting data in MarkLogic

Why stop there? The application can do more by displaying extra system information and saving the utilisation metrics to a database for later retrieval.

Back in the Node.js code, let’s collect some additional metrics and feed them to the handlebars template:

let dataObject = {
  osType: os.type().toLowerCase() === 'darwin' ? 'Mac OS X' : os.type(),
  osReleaseVersion: os.release(),
  osArch: os.arch(),
  osCPUs: os.cpus(),
  osHostname: os.hostname(),
  osTotalMemory: Number(os.totalmem() / 1073741824).toFixed(0),
};
// etc
let indexRoute = (req, res) => {
  res.render(__dirname + '/index', { data: dataObject });
};

Let’s also persist this data by adding a few lines to the Node.js application:

db.documents
  .write({
    uri: '/data/host.json',
    contentType: 'application/json',
    content: dataObject,
  })
  .result()
  .then((response) => {
    console.log(response.documents[0].uri + ' inserted to the database.');
  })
  .catch((error) => {
    console.log(error);
  });

If you’d like to know how to connect to the MarkLogic database and learn more about the Node.js Client API please read this article.

The CPU and memory metrics should be persisted too, but the question is how. Ideally, one document gets created per data collection. That means modifying socket.js (where the data collection happens). The new setInterval() looks like this:

setInterval(() => {
  load = os.loadavg()[0];
  freeMemory = os.freemem();
  usedMemory = Number((totalMemory - freeMemory) / 1073741824).toFixed(4);
  socket.emit('resources', { cpu: load, memory: usedMemory });
  db.documents
    .write({
      uri: '/data/' + Date.now() + '.json',
      contentType: 'application/json',
      content: { cpu: load, memory: usedMemory },
    })
    .result()
    .then((response) => {
      console.log(response.documents[0].uri + ' inserted to the database');
    })
    .catch((error) => {
      console.log(error);
    });
}, 5000);

Documents are stored with the URI (unique document identifier) of /data/EPOCH.json. (If you’re not sure how URIs work in MarkLogic, refer to the article I linked earlier.)

All that’s left is displaying the data in the handlebars template:

<html>
  <head>
    <script src="/socket.io/socket.io.js"></script>
    <script
      type="text/javascript"
      src="https://www.gstatic.com/charts/loader.js"
    ></script>
    <script type="text/javascript" src="/js/charting.js"></script>
    <script
      type="text/javascript"
      src="/bower_components/jquery/dist/jquery.js"
    ></script>
    <link
      rel="stylesheet"
      href="/bower_components/bootstrap/dist/css/bootstrap.css"
    />
  </head>
  <body>
    <div class="content">
      <div id="error"></div>
      <h1>System Information</h1>
      <div class="panel panel-default">
        <div class="panel-heading">
          <h3 class="panel-title">
            {{ data.osHostname }} - {{ data.osType }} ({{ data.osArch }}) ({{
            data.osReleaseVersion }})
          </h3>
        </div>
        <div class="panel-body">
          <p>
            <strong>System CPU</strong>: {{ data.osCPUs.0.model }},
            <strong>Total memory</strong>: {{ data.osTotalMemory }}GB
          </p>
          <div id="curve_chart" style="width: 900px; height: 500px"></div>
        </div>
      </div>
    </div>
  </body>
</html>

Conclusion

The application is now complete. It collects CPU and memory utilisation every 5 seconds and saves that data to the MarkLogic database with the same structure discussed earlier.

In a later article we’ll look at how to run ‘historical’ reporting on the collected data using documents in the database. Stay tuned!