I’ve delivered a talk at a few conferences and meetups titled “Supercharge your JavaScript with Web Assembly.” As part of that talk, I showcased a C++ project ported to the web using web assembly.

Web assembly lets us bring native applications to the web. There are three main reasons you’d want to consider it:

• Reuse existing code: you already have something written in C or Rust and you want to bring it to the web.
• Performance: most of the time JavaScript and web assembly perform similarly, but web assembly has more predictable performance due to how the V8 engine creates machine-readable code.
• Binary size: a web assembly module can be compressed heavily using gzip or brotli, and can end up much smaller than a JavaScript file.

The use-case we’ll walk through here relates to the first point.

Back in 2017, my colleague Jon Sneyers created a C++ project called SSIMULACRA, “Structural SIMilarity Unveiling Local And Compression Related Artifacts.”

You can learn more about the project, and the implementation details of SSIMULACRA in the article titled Detecting the psychovisual impact of compression related artifacts using SSIMULACRA by Jon.

In a nutshell, it compares two images and detects quality differences between them. Image compression and optimisation are critical to web performance, but optimisation shouldn’t destroy visual fidelity. You don’t want to over-optimise an image and produce a low-quality version. SSIMULACRA compares two versions of the same image and tells you if one is over-optimised. The result is a number: closer to 1 means a really pixelated image, closer to 0 means the image is optimised well with no visible defects.

Determining the “right” quality for an image is a tricky task. As per the linked article: “If the value is above 0.1 (or so), the distortion is likely to be perceptible/annoying.”

Eric Portis, another colleague of mine, has done some exhaustive research around this very subject; you can read his findings published in the article Human-redable image quality scores.

The original project requires compiling and executing a C++ binary, passing two images as arguments via the CLI: ./ssimulacra img1.jpg img2.jpg.

I find this tool brilliant, but I reckoned it would bring more value as a web application where you could also see the images being compared side by side. How do you port a C++ project to the web? Web assembly.

Getting started

Before the details, let’s cover the environment. For this port I used a virtualised environment running Ubuntu (v18.04.2, verifiable by running lsb_release -a) on VirtualBox. I’d recommend configuring the OS for SSH and file sharing via Samba for easy access.

SSIMULACRA depends on OpenCV, so that’s where we start.

OpenCV

“OpenCV (Open Source Computer Vision Library) is an open-source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilise and modify the code.”

Multiple versions of OpenCV exist. At the time of writing, v4 is the latest, and that’s what we’ll install. Since we’ll need some customisation later, we’ll build from source.

Update the package repository, update your packages, and install the build dependencies:

$ sudo apt-get update && sudo apt-get upgrade -y
$ sudo apt-get install build-essential -y
$ sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev -y

(If you get an error that some of the lib* packages are not available, run the command be;pw first, and then rerun the previous few commands.

$ sudo add-apt-repository main && sudo add-apt-repository universe &&  sudo add-apt-repository restricted && sudo add-apt-repository multiverse

These are optional but recommended:

$ sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev -y

Now we’re ready for the OpenCV build. Navigate to your preferred folder (I’m using /root) and run:

$ mkdir ~/src
$ cd ~/src
$ git clone https://github.com/opencv/opencv.git
$ cd opencv
$ mkdir build && cd build

This clones the latest OpenCV repository and creates a build folder. It’s customary to separate src and build folders when working with CMake.

Inside the build folder, we start compilation. Specific flags need to be enabled; miss them and you’ll be repeating the process. The make step later can take 15 to 30 minutes.

$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
  -D CMAKE_INSTALL_PREFIX=/usr/local \
  -D INSTALL_PYTHON_EXAMPLES=ON \
  -D OPENCV_GENERATE_PKGCONFIG=YES \
  -D INSTALL_C_EXAMPLES=ON ..

Some options above are not necessarily required (like the examples), feel free to change those around.

For the OPENCV_GENERATE_PKGCONFIG option, you need to pass YES, not ON. (Yes, really.) The C and Python examples aren’t required but can be handy as reference.

If CMake reports no errors, run:

$ make -j$(nproc)

Go brew a coffee. The build process takes a while.

Once make finishes successfully, verify the installation with:

$ pkg-config --cflags opencv4 # get the include path (-I)
$ pkg-config --libs opencv4  # get the libraries path (-L) and all the libraries (-l)

The first should return -I/usr/local/include/opencv4. The second should return -L/usr/local/lib -lopencv_dnn -lopencv_gapi -lopencv_highgui -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_video -lopencv_calib3d -lopencv_features2d -lopencv_flann -lopencv_videoio -lopencv_imgcodecs -lopencv_imgproc -lopencv_core. These are the paths for the OpenCV core files and libraries.

With OpenCV installed, we can set up SSIMULACRA itself. Clone it from GitHub:

cd /root && git clone https://github.com/cloudinary/ssimulacra && cd ssimulacra

If you try to compile now, it’ll fail:

$ make
ssimulacra.cpp:81:10: fatal error: cv.hpp: No such file or directory
 \#include <cv.hpp>
​     ^~~~~~~~

Expected. A few changes are needed in ssimulacra.cpp since it was built for an older version of OpenCV:

• replace #include <cv.hpp> with #include <opencv2/opencv.hpp>
• remove #include <highgui.h>

The latest version of OpenCV uses opencv.hpp, and there’s also no need to add highgui.h since the imread method has been moved to the OpenCV core.

Note that since the writing of this article, the original SSIMULACRA repository has been updated to reflect the changes above.

The Makefile also needs editing since it references opencv instead of opencv4. Here’s the updated version:

CFLAGS=`pkg-config --cflags opencv4`
LDFLAGS=`pkg-config --libs opencv4` -lopencv_core -lopencv_imgcodecs -lopencv_imgproc

Run make again:

$ make

If there are no errors, you should have an executable called ssimulacra. Running it without arguments throws a warning, but confirms the binary works:

$ ./ssimulacra
./ssimulacra
Usage: ./ssimulacra orig_image distorted_image
Returns a value between 0 (images are identical) and 1 (images are very different)
If the value is above 0.1 (or so), the distortion is likely to be perceptible/annoying.
If the value is below 0.01 (or so), the distortion is likely to be imperceptible.

If you have two images available (same image, different qualities), run the binary: ./ssimulacra image1.jpg image2.jpg.

SSIMULACRA to Web Assembly

With OpenCV and SSIMULACRA set up, we can port to web assembly. There are quite a few steps because the application relies on libraries we can’t bring directly to the “wasm” world. We first need a web assembly build of OpenCV, then use those resulting libraries when building the wasm file via Emscripten.

Setup Emscripten

Emscripten is a toolchain for compiling to asm.js and WebAssembly, built using LLVM. It lets you run C and C++ on the web at near-native speed without plugins.

Set it up via GitHub:

$ cd /root && git clone https://github.com/emscripten-core/emsdk.git && cd emsdk
$ ./emsdk install latest
$ ./emsdk activate latest
$ source ./emsdk_env.sh

Note that source ./emsdk_env.sh will need to be invoked whenever a new terminal session is started.

Emscripten is now installed.

Next: the web assembly OpenCV build. This is the tricky part because we need to enable all the options SSIMULACRA requires. Navigate to and open the build file:

$ cd /root/src/opencv/platforms/js
$ vi build_js.py

Make the following changes (leave the rest unchanged):

-DWITH_JPEG=ON
-DWITH_WEBP=ON
-DWITH_PNG=ON
-DWITH_TIFF=ON
-DBUILD_ZLIB=ON,
-DBUILD_opencv_apps=OFF,
-DBUILD_opencv_calib3d=OFF,
-DBUILD_opencv_dnn=ON,
-DBUILD_opencv_features2d=OFF,
-DBUILD_opencv_flann=ON,
-DBUILD_opencv_gapi=OFF,
-DBUILD_opencv_ml=OFF,
-DBUILD_opencv_photo=OFF,
-DBUILD_opencv_imgcodecs=ON,

Execute the build:

$ python /root/src/opencv/platforms/js/build_js.py build_wasm --build_wasm --emscripten_dir="/root/emsdk/upstream/emscripten"

Another coffee break. This takes 15 to 30 minutes.

Sometimes the build process doesn’t compile all selected libraries. To verify, run:

$ { find /root/src/build_wasm/3rdparty/lib -type f -name "*.a" | wc -l & find /root/src/build_wasm/lib -type f -name "*.a" | wc -l } | cat

You should get 6 and 10 respectively. If the values differ, navigate into the two subfolders (build_wasm/3rdparty/lib and build_wasm/lib) and check for these .a files:

$ pwd && l
/root/src/opencv/build_wasm/3rdparty/lib
total 4.9M
drwxr-xr-x 2 root root 4.0K Jun 20 20:53 .
drwxr-xr-x 9 root root 4.0K Jun 20 20:16 ..
-rw-r--r-- 1 root root 398K Jun 20 20:39 liblibjpeg-turbo.a
-rw-r--r-- 1 root root 233K Jun 20 20:40 liblibpng.a
-rw-r--r-- 1 root root 3.3M Jun 20 20:20 liblibprotobuf.a
-rw-r--r-- 1 root root 478K Jun 20 20:41 liblibtiff.a
-rw-r--r-- 1 root root 466K Jun 20 20:42 liblibwebp.a
-rw-r--r-- 1 root root  96K Jun 20 20:16 libzlib.a

$ pwd && l
/root/src/opencv/build_wasm/lib
total 17M
drwxr-xr-x  2 root root 4.0K Jun 20 20:45 .
drwxr-xr-x 14 root root 4.0K Jun 20 20:16 ..
-rw-r--r--  1 root root 2.1M Jun 20 20:27 libopencv_calib3d.a
-rw-r--r--  1 root root 2.6M Jun 20 20:20 libopencv_core.a
-rw-r--r--  1 root root 5.5M Jun 20 20:29 libopencv_dnn.a
-rw-r--r--  1 root root 718K Jun 20 20:25 libopencv_features2d.a
-rw-r--r--  1 root root 555K Jun 20 20:21 libopencv_flann.a
-rw-r--r--  1 root root 488K Jun 20 20:45 libopencv_imgcodecs.a
-rw-r--r--  1 root root 3.6M Jun 20 20:23 libopencv_imgproc.a
-rw-r--r--  1 root root 418K Jun 20 20:29 libopencv_objdetect.a
-rw-r--r--  1 root root 790K Jun 20 20:24 libopencv_photo.a
-rw-r--r--  1 root root 325K Jun 20 20:29 libopencv_video.a

You might be missing libopencv_imgcodecs.a and the image codec libs like liblibpng.a.

To fix this, open the 3rdparty folder. You should see a folder for each image codec (libwebp, etc.). Navigate into each and run make:

$ cd /root/src/opencv/build_wasm/3rdparty/libwebp && make

If libopencv_imgcodecs.a is missing, navigate to the modules folder and run make:

$ cd /root/src/opencv/build_wasm/modules/imgcodecs && make

Make sure the right flags are set in build_js.py as described earlier.

Create the wasm file

Run emcc to create the wasm file. Before that, we need to edit the SSIMULACRA source code. Make these changes:

# Include two more header files:
#include <opencv2/imgcodecs.hpp>
#include <emscripten/emscripten.h>

We’ll create a dedicated function for the SSIMULACRA calculation and rework the C++ main() function. Replace int main(int argc, char **argv) with int calc(). Strip out the first few printf instructions (they’re not needed) and replace the opening lines with:

img1_temp = imread("image1.ext",-1);
img2_temp = imread("image2.ext",-1);
int nChan = img1_temp.channels();
if (nChan != img2_temp.channels()) {
  fprintf(stderr, "Image file image1 has %i channels, while\n",  nChan);
  fprintf(stderr, "image file image2 has %i channels. Can't compare.\n", img2_temp.channels());
  return -1;
}

This ensures the right files are being read (more on this later) and removes all references to argv.

Prepend int calc() with EMSCRIPTEN_KEEPALIVE, which requires wrapping it in extern "C" {} to make the keyword available. (extern "C" gives a C++ function name C linkage.)

Here’s the trimmed calc() function:

#ifdef __cplusplus
extern "C"
{
#endif
 EMSCRIPTEN_KEEPALIVE
 int calc()
 {
  Scalar sC1 = {C1, C1, C1, C1}, sC2 = {C2, C2, C2, C2};
  Mat img1, img2, img1_img2, img1_temp, img2_temp, img1_sq, img2_sq, mu1, mu2, mu1_sq, mu2_sq, mu1_mu2, sigma1_sq, sigma2_sq, sigma12, ssim_map;
  // read and validate input images
  img1_temp = imread("image1.ext", -1);
  img2_temp = imread("image2.ext", -1);

  int nChan = img1_temp.channels();
  if (nChan != img2_temp.channels())
  {
   fprintf(stderr, "Image file image1 has %i channels, while\n", nChan);
   fprintf(stderr, "image file image2 has %i channels. Can't compare.\n", img2_temp.channels());
   return -1;
  }
# at the end of calc():
#ifdef __cplusplus
}
#endif

We still need a main() function, but keep it minimal:

int main(int argc, char **argv) {
 return 0;
}

Now we can compile SSIMULACRA to Web Assembly. Open Makefile and add:

ssimulacra: ssimulacra.cpp
  emcc -std=c++11 -O3 -fstrict-aliasing -ffast-math -I/usr/local/include/opencv4 -L/root/src/opencv/build_wasm/lib -lopencv_core -lopencv_imgcodecs -lopencv_imgproc -L/root/src/opencv/build_wasm/3rdparty/lib -llibjpeg-turbo -llibpng -llibwebp -llibtiff -llibzlib -llibprotobuf -s LLD_REPORT_UNDEFINED ssimulacra.cpp -s WASM=1 -s MODULARIZE -s EXPORT_NAME="WAModule" -s EXPORTED_RUNTIME_METHODS='["FS", "ccall"]' -s FORCE_FILESYSTEM=1 -s NO_EXIT_RUNTIME=1 -s ALLOW_MEMORY_GROWTH=1 -o ssimulacra.js

Make sure that the paths are correctly set for the generated lib files.

Run emmake make. It takes about 30 to 60 seconds. At the end, you should see two new files: ssimulacra.js and ssimulacra.wasm. These are the two files needed for a web application.

A quick run through the options: emcc is the Emscripten compiler that builds the Web Assembly file. We’re making the Emscripten file system available and modularising the JavaScript output (so we can import the Web Assembly file like a standard ES2015 module). NO_EXIT_RUNTIME and ALLOW_MEMORY_GROWTH are required so we can keep calling calc() from our web app without worrying about memory allocation.

Using ssimulacra.wasm

Copy both the .js and .wasm files to your web application and bolt on some simple HTML (note the script inclusion):

<input
  id="image1"
  placeholder="https://res.cloudinary.com/tamas-demo/image/upload/jam/darthvader.jpg"
/>
<input
  id="image2"
  placeholder="https://res.cloudinary.com/tamas-demo/image/upload/q_auto/jam/darthvader.jpg"
/>
<button id="calculate">SSIMULACRA</button>
<span id="score"></span>

<script src="ssimulacra.js"></script>

We don’t need to import the Web Assembly file separately. Emscripten’s generated JavaScript file handles the import automatically.

Pro tip: To use Web Assembly streaming, for faster load times, the HTTP server that you use should have the appropriate Content-Type setup for wasm files. Here’s an example HTTP server created in Python:

# server.py
import http.server
from http.server import HTTPServer, BaseHTTPRequestHandler
import socketserver

PORT = 8999

Handler = http.server.SimpleHTTPRequestHandler

Handler.extensions_map = {
  '.html': 'text/html',
  '.png': 'image/png',
  '.jpg': 'image/jpg',
  '.wasm': 'application/wasm',
  '.css': 'text/css',
  '.js': 'application/x-javascript',
  '': 'application/octet-stream',
}

httpd = socketserver.TCPServer(("", PORT), Handler)

print("serving at port", PORT)
httpd.serve_forever()

Call this server by executing python server.py.

Now let’s wire up SSIMULACRA. We need to read the two Cloudinary URLs, transform them into a Uint8Array, and save that in memory for Web Assembly. We use the Fetch API and its arrayBuffer() method to get a buffer and convert it to a Uint8Array.

const oneInput = document.getElementById('one');
const twoInput = document.getElementById('two');
const score = document.getElementById('score');

const urlToUint8Array = async (url) => {
  const response = await fetch(url);
  const buffer = await response.arrayBuffer();
  const arr = new Uint8Array(buffer);
  return arr;
};

Instantiate the Web Assembly module. We used the MODULARIZE option when running emcc, plus EXPORT_NAME="WAModule", which is the module name we can import:

document.addEventListener('DOMContentLoaded', async () => {
  const waModule = await WAModule();
  // waModule now has access to the Web Assembly file system
  // as well as all the exported functions such as calc()
  // ... the rest of the code goes here

With the helper method and module ready, write the two files to Web Assembly memory and call calc():

document.getElementById('calculate').addEventListener('click', async () => {
  const img1 = await urlToUint8Array(image1.value);
  const img2 = await urlToUint8Array(image2.value);
  waModule.FS.writeFile('image1.ext', img1);
  waModule.FS.writeFile('image2.ext', img2);
  const ssimulacraScore = waModule.ccall('calc', 'int', [], null);
  score.innerText = ssimulacraScore;
});

Note that FS.writeFile() writes to memory and creates a binary file by default. Remember those changes we made in ssimulacra.cpp, ensuring these are the two files being read for the calculation.

Open the web application, add two images, hit the SSIMULACRA button. If everything is wired up correctly, the score should appear.

An interesting addition: showing the two images side by side. One approach is obvious (reading the Cloudinary URL from the input box). The other is less obvious: pull the binary file from Web Assembly memory and rebuild the image from it:

// just add <div id="img"></div> to the HTML
const array = waModule.FS.readFile('image1.ext');
const base64Data = btoa(String.fromCharCode.apply(null, array));
const img = new Image();
img.src = `data:image/jpg;base64,${base64Data}`;
document.getElementById('img').appendChild(img);

Conclusion

This project was a challenge, but a rewarding one. One of the strongest use-cases for Web Assembly is porting existing applications to the web. The example outlined here should give you a clear picture of what that involves and why it’s worth the effort.