Bridging Python and Matlab: Passing callbacks and optimization

This blog post is about a bridge I created that allows you to pass callbacks to Python functions directly to Matlab code without having to resort to things like sockets or COM.

Background

Designing our UAVs involves a number of different codes that interact & must be coordinated to solve what is essentially a Multidisciplinary Design Optimization (MDO) problem.  One of those codes is written in Python and involves a constrained optimization problem of the following kind:

Find the minimum of a problem specified by

xbbeqlb, and ub are vectors, A and Aeq are matrices, c(x) and ceq(x) are functions that return vectors, and f(x) is a function that returns a scalar. f(x), c(x), and ceq(x) can be nonlinear functions.

In our case we have 10 variables (x has length 10), 28 nonlinear constraint functions c(x), 4 nonlinear equality constraint functions ceq(x) and box constraints on most variables.  The objective function f(x) and all constraint functions are pure python functions.

Thus to optimize f(x) you need an optimizer that can handle python functions, and for that there is the excellent openopt library.  This is what I did and after trying out the different NLP solvers I settled on the scipy_slsqp solver for most runs as it seemed to consistently give the best results.

However, as the optimization problem was tweaked and as we did more runs, we did run into cases where it would not solve and the aeronautical engineers I work with were not always happy with its robustness.  Also nobody had ever worked with the openopt library but everybody knew and trusted the various Matlab solvers.  Besides, us being part of the Computational Engineering Design Group, a number of powerful in-house Matlab solvers had been developed and were available to me.  Thus, the least I could do was to try to use those and see if that improved anything.

Python-Matlab bridge

So I started searching the web for tools / libraries that allowed me to bridge python/Matlab, specifically tools that supported passing callbacks between the two languages.  A promising project seemed mlabwrap but while it was great for evaluating Matlab expressions,  it does (at the time of writing) not support callbacks.

In the end I decided to see if I could scratch my own itch.  I really did not want to deal with sockets or any kind of network protocol and I also wanted the solution to be crossplatform, so no COM.  Since both Python and Matlab have well documented interfaces with C/C++ code, that seemed to be the way to go.

So I sat down with the ctypes & mex documentation and with a little help from Mathworks support was able to cobble together a simple example using fminsearch.  By the way, I had never done any Python – C work before, but ctypes is awesome and a joy to use.  The little trick to make it all work is that you just have to treat the function handle as an integer (it’s just a memory address after all).  Then all you are really doing is passing an integer around which is straightforward.

With fminsearch working the goal was to extend to fmincon since that would give us all the basic functionality needed.  This took a bit of debugging and deciphering Matlab crash dumps but I finally got it working.  The final flow of events looks like this:

You start off with writing a matlab file that wraps the optimizer you are interested in (in this case fmincon) and exposes its signature.  This m file forwards all function handle arguments to a mex file which converts the integer to a function pointer and calls it.  This is how we get the callback to a C function.  A simple bridge library is then needed which is loaded by ctypes and converts the arguments into something the Matlab shared library file understands.

From my initial tests I see it is finding the same optima as scipy_slsqp was using openopt. Now I just need to do more extensive testing to see if performance is any more robust. After that I can start looking into the other solvers provided by Matlabs optimization toolboxes as well as our own solvers.

Source code

All the code is available on github.  I hope it is useful to somebody and I’d be very happy to receive patches or improvements.  I guess this may even be of use to the mlabwrap / openopt projects.

–Dirk

Using Javaprop in .net

JavaProp is tool for designing and analyzing propellers.  From the website:

JavaProp is a relatively simple program, which is based on the blade element theory. The blade is divided into small sections, which are handled independently from each other. Each segment has a chord and a blade angle and associated airfoil characteristics. The theory makes no provision for three dimensional effects, like sweep angle or cross flow. But it is able to find the additional axial and circumferential velocity added to the incoming flow by each blade segment. This additional velocity results in an acceleration of the flow and thus thrust. Usually this simplified model works very well, when the power and thrust loading of the propeller (power per disk area) is relatively small, as it is the case for most aircraft propellers.

It comes in the form of an applet that you can use through the website or as a jar file that you can link into your application.  While it seems like a small and simple program its actually quite powerful and get you a long way towards picking/sizing a propeller for your aircraft.

At some point there was the idea to integrate into Excel as a plugin that would link into our sizing equations.  Turns out that writing a java plugin for excel is actually a rather messy business and I really did not want to go down that route.  Instead I had the idea of trying to convert the java bytecode to bytecode that runs on the CLR.  This would make things a lot easier as the excel plugin can be written in C#, using all the bells and whistles that visual studio gives you.

Some googling led me to IKVM and I have to say I was very pleasently surprised. I was expecting lots of strange errors but everything worked smoothly and out of the box.  It generated a dll which I could test on linux using mono as well as with Visual Studio on Windows. The excel plugin never materialized however, since the control flow JavaProp assumed did not match the flow used in our sizing equations but it was still fun to mess around with.  In case anybody wants to do the same simply downlad the JavaProp jar as well as the mhclasses dependency and use the following ikvm command to convert it to a dll:

ikvmc -target:library javaprop.jar mhclasses.jar

To include it in a .Net application you just have to make sure you also add the reference to IKVM.OpenJDK.Core.dll.

–Dirk