Wrapping dlib for Matlab

dlib is a famous c++ machine learning library and Matlab is a numeric computing platform that provide many algorithms and tools to do researches. There are lot of ML algorithms in dlib that Matlab doesn't have. But still we can use the dlib algorithms in Matlab by compiling the dlib functions as Mex files. In this post I am discussing about the official approach to make a dlib wrapper to matlab.

Requirements

• Matlab
• dlib (dlib should be installed in the computer and the path to dlib header files should be in the include path)

Folder Structure

- +dlib/
- -- private/       # C++ source files
- -- build/         # Makefiles that generated by cmake
- -- .gitignore     # List of ignored file names and folder names.
- -- *.m            # Matlab classes linked to Mex files
- -- *.mexa64       # Built mex files
- demo.m            # Matlab file to test the wrapper

Getting Started

You can find an example dlib wrapper in dlib/matlab folder of the dlib Github repository. Copy the following files from this folder to the +dlib/private folder of your workspace.

CMakeLists.txt
cmake_mex_wrapper
mex_wrapper.cpp
call_matlab.h

Change the CMakeLists.txt to find dlib from the include path instead from an absolute path.

# +dlib/private/CMakeLists.txt

# Remove the following line
add_subdirectory(.. dlib_build)
# And add the following line
find_package(dlib REQUIRED)

And remove following lines from the CMakeLists.txt to start our project as a new one.

# +dlib/private/CMakeLists.txt

add_mex_function(example_mex_function dlib::dlib)
add_mex_function(example_mex_callback dlib::dlib)
add_mex_function(example_mex_struct dlib::dlib)
add_mex_function(example_mex_class dlib::dlib)

By default the built Mex file is saving to the source directory. But our source directory is a separate folder named private. So we need to save the Mex file to the parent directory of the private folder. Change CMakeLists.txt as following to change it.

# +dlib/private/CMakeLists.txt

// ...
# You can tell cmake where to put the mex files when you run 'make install' by
# setting this variable.  The path is relative to this CMakeLists.txt file.
get_filename_component(MEX_INSTALL_PATH "${CMAKE_CURRENT_SOURCE_DIR}/../" ABSOLUTE)
set(install_target_output_folder ${MEX_INSTALL_PATH})

If you are using linux, change the value of MATLAB_EXECUTABLE in the cmake_mex_wrapper file.

# +dlib/private/cmake_mex_wrapper

find_program(MATLAB_EXECUTABLE matlab PATHS
        "/usr/local/MATLAB/*/bin"
        )

We are going to build the dlib example project as a standalone project. But it made to compile as a component in dlib project. So there are some relative includes in the mex_wrapper.cpp file. Change all of these relative includes as the following.

# +dlib/private/mex_wrapper.cpp

// ...

// Change following lines
#include "../matrix.h"
#include "../array2d.h"
#include "../array.h"
#include "../image_transforms.h"
#include "../is_kind.h"
#include "../string.h"
#include "../any.h" // for sig_traits
#include "../hash.h"

// As in the following
#include <dlib/matrix.h>
#include <dlib/array2d.h>
#include <dlib/array.h>
#include <dlib/image_transforms.h>
#include <dlib/is_kind.h>
#include <dlib/string.h>
#include <dlib/any.h> // for sig_traits
#include <dlib/hash.h>

And add the dlib namespace to the mex_wrapper.cpp file.

# +dlib/private/mex_wrapper.cpp

// All includes goes here
//..

using namespace dlib;

Creating a C++ class and compile

As the start create a C++ file to implement the wrapper to your dlib function. In this example I am implementing a wrapper to the dlib CNN face detector. Because it is more accurate than the Matlab built-in face detector. I am creating a C++ class file and copying the contents of the dlib/matlab/examplemexclass.cpp file to my new class file. And I am removing the inner contents and comments from the file.

# +dlib/private/CNNFaceDetectorMex.cpp

#include <iostream>

using namespace std;

class CNNFaceDetector 
{
public:

    CNNFaceDetector()
    {
    }

    void test(){
      cout << "Yeah! It worked";
    }
};

#define MEX_CLASS_NAME CNNFaceDetector 
#define MEX_CLASS_METHODS test


#include "mex_wrapper.cpp"

I am including it to the CMakeLists.txt to compile as a Mex file.

# +dlib/private/CMakeLists.txt

# ...

add_mex_function(CNNFaceDetectorMex dlib::dlib)

Create an another CMakeLists.txt file in the root folder to tell cmake to compile the +dlib/private folder.

# CMakeLists.txt

add_subdirectory(+dlib/private)

Now compile the project using following commands.

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=../
$ make
$ make install

You can find the compiled Mex file in the +dlib folder.

Creating A Matlab class to wrap Mex file

The Mex files are not classes. These files are acting like functions. But we can convert Mex files to classes by wrapping the Mex file in a Matlab class.

Run the Mex file as a function in the Matlab command line to generate the contents of the Matlab class file.

>> dlib.CNNFaceDetectorMex()

This command is generating the contents that should be in the Matlab class file. Copy this contents to your matlab class file.

# +dlib/CNNFaceDetector.m

classdef CNNFaceDetector < handle
    properties (Access = private)
        cpp_ptr
    end

    methods
        function this = CNNFaceDetector()
            this.cpp_ptr = CNNFaceDetectorMex('construct');
        end

        function copied_obj = clone(this)
            %Returns a new independent object that is a copy of this.
            copied_obj = CNNFaceDetector();
            copied_obj.cpp_ptr = CNNFaceDetectorMex(this.cpp_ptr,'clone');
        end

        function varargout = test(this, varargin) 
            [varargout{1:nargout}] = CNNFaceDetectorMex(this.cpp_ptr, 1, varargin{:}); 
        end 

    end

    methods(Access=private) 
        function delete(this) 
            CNNFaceDetectorMex(this.cpp_ptr); 
        end         
    end 

end

Now create an instance of this class in the demo.m file and try to run it.

# demo.m

detector = dlib.CNNFaceDetector();
detector.test()

I got the following error by running the script.

>> demo
Warning: The following error was caught while executing 'dlib.CNNFaceDetector' class destructor:
Undefined function 'CNNFaceDetectorMex' for input arguments of type 'double'.

Error in dlib.CNNFaceDetector/delete (line 25)
            CNNFaceDetectorMex(this.cpp_ptr);

Error in demo (line 1)
detector = dlib.CNNFaceDetector(); 
> In demo (line 1) 
'CNNFaceDetectorMex' is not found in the current folder or on the MATLAB path, but exists in:
    /home/ramesh/Researches/EyeTouch/facetrack/+dlib

Change the MATLAB current folder or add its folder to the MATLAB path.

Error in dlib.CNNFaceDetector (line 8)
            this.cpp_ptr = CNNFaceDetectorMex('construct');

Error in demo (line 1)
detector = dlib.CNNFaceDetector();

This error means the CNNFaceDetectorMex function is not in the current namespace. But it is available in the dlib namespace. So I changed the all CNNFaceDetectorMex words to dlib.CNNFaceDetectorMex in the CNNFaceDetector.m file.

# +dlib/CNNFaceDetector.m

classdef CNNFaceDetector < handle
    properties (Access = private)
        cpp_ptr
    end

    methods
        function this = CNNFaceDetector()
            this.cpp_ptr = dlib.CNNFaceDetectorMex('construct');
        end

        function copied_obj = clone(this)
            %Returns a new independent object that is a copy of this.
            copied_obj = CNNFaceDetector();
            copied_obj.cpp_ptr = dlib.CNNFaceDetectorMex(this.cpp_ptr,'clone');
        end

        function varargout = test(this, varargin) 
            [varargout{1:nargout}] = dlib.CNNFaceDetectorMex(this.cpp_ptr, 1, varargin{:}); 
        end 

    end

    methods(Access=private) 
        function delete(this) 
            dlib.CNNFaceDetectorMex(this.cpp_ptr); 
        end         
    end 

end

Finally I could successfully run the demo.m.

>> demo
Yeah! It worked>>

Do not forget to repeat this step in each time you changed the Mex class methods.

Putting dlib algorithm

As in this example I downloaded the trained dataset and extracted the .dat file to data folder.

After I created a network in the CNNFaceDetectorMex.cpp file as in the above example.

# +dlib/private/CNNFaceDetectorMex.cpp

// ...
#include <dlib/dnn.h>
// ...
using namespace dlib;

template <long num_filters, typename SUBNET>
using con5d = con<num_filters, 5, 5, 2, 2, SUBNET>;
template <long num_filters, typename SUBNET>
using con5 = con<num_filters, 5, 5, 1, 1, SUBNET>;

template <typename SUBNET>
using downsampler = relu<affine<
    con5d<32, relu<affine<con5d<32, relu<affine<con5d<16, SUBNET>>>>>>>>>;
template <typename SUBNET> using rcon5 = relu<affine<con5<45, SUBNET>>>;

using net_type = loss_mmod<
    con<1, 9, 9, 1, 1,
        rcon5<rcon5<
            rcon5<downsampler<input_rgb_image_pyramid<pyramid_down<6>>>>>>>>;

class CNNFaceDetector 
{
// ...
private:
  net_type _net;
}

Next implemented a new method to train the detector and initialize it. deleted the old method train

# +dlib/private/CNNFaceDetector.cpp

// ...

#include <dlib/image_processing/full_object_detection.h>

// ...

class CNNFaceDetector
{
public:
// ...
  void train(string fd_file_name) { deserialize(fd_file_name) >> _net; }
}

// ...
#define MEX_CLASS_METHODS train

I compiled these changes again and repeated the Mex class creating step. Then I edited the demo.m as following.

# demo.m

detector = dlib.CNNFaceDetector();
detector.train('./data/mmod_human_face_detector.dat');

It executed without any error.

Next I implemented a method to do the facial detection. This method is taking a RGB picture as an input and Returning a list of bounding boxes of detected faces.

# +dlib/private/CNNFaceDetectorMex.cpp

// ...

#include <dlib/matrix/matrix.h>
#include <dlib/pixel.h>

// ...

class CNNFaceDetector
{
public:
// ...

  void detect(const matrix<rgb_pixel> &img,
              std::vector<matrix_colmajor> &bboxes) {

    std::vector<mmod_rect> rects = _net(img);

    if (rects.size() > 0) {
      matrix_colmajor bbox = matrix_colmajor();
      bbox.set_size(1, 4);
      for (int i = 0; i < rects.size(); i++) {
        mmod_rect rect = rects[i];
        long t = rect.rect.top();
        long l = rect.rect.left();
        bbox(0, 0) = l;
        bbox(0, 1) = t;
        bbox(0, 2) = rect.rect.right() - l;
        bbox(0, 3) = rect.rect.bottom() - t;
        bboxes.push_back(bbox);
      }
    }
  }
// ...
}

// ...
#define MEX_CLASS_METHODS train, detect

After compiling it I repeated the Mex class creating step again.

And I downloaded an image to the input folder and changed the demo.m file as following.

# demo.m

% ...
I = imread("./input/students.jpg");
faces = detector.detect(I);

After running the above script I got the following error.

>> demo
Error using CNNFaceDetectorMex
mex_function has some bug in it related to processing input argument 1

Error in dlib.CNNFaceDetector/detect (line 22)
            [varargout{1:nargout}] = dlib.CNNFaceDetectorMex(this.cpp_ptr, 2, varargin{:});

Error in demo (line 5)
faces = detector.detect(I);

This error is causing because there is no method to cast a Matlab image to matrix<rgb_pixel>. I implemented a new method in the mex_wrapper.cpp to cast a Matlab image to dlib matrix.

# +dlib/private/mex_wrapper.cpp

// After line 640
    template <
        long num_rows,
        long num_cols,
        typename mem_manager,
        typename layout
        >
    void assign_image (
        const long arg_idx,
        matrix<dlib::rgb_pixel,num_rows,num_cols, mem_manager,layout>& img,
        const dlib::uint8* data,
        long nr,
        long nc
    )
    {

        img.set_size(nr, nc);
        for (long c = 0; c < img.nc(); ++c)
            for (long r = 0; r < img.nr(); ++r)
                img(r,c).red = *data++;
        for (long c = 0; c < img.nc(); ++c)
            for (long r = 0; r < img.nr(); ++r)
                img(r,c).green = *data++;
        for (long c = 0; c < img.nc(); ++c)
            for (long r = 0; r < img.nr(); ++r)
                img(r,c).blue = *data++;
    }

After compiling it, the matlab script was executed without any errors.

Again I changed the matlab script to see a graphical output of the result.

# demo.m

% ...

for i = 1:length(faces)
    bbox = cell2mat(faces(i));
    I = insertShape(I, 'Rectangle', bbox, 'LineWidth',5);
end

figure; imshow(I); title('Detected faces');

It gives me the following output.