Data for Good ?

pydatameetup

A short post to share the slides I used during my talk at the 4th PyData London Meetup. Organized by Ian Oszvald, co-Taarifian Florian Rathgeber, and others, its always a full house with a friendly crowd.

Given the limited time I had to put something together I think the talk was well received and triggered lots of good feedback and conversations in the pub afterwards. Hopefully I managed to ‘turn’ a few attendees ūüôā

Aside: There has been a lot happening on my end the last couple months and its been tough (but exciting!) keeping up with it all. My digital prescense has been lagging behind somewhat but slowly getting there.

–Dirk

 

PyData London: Modeling Departures from Routine in Human Mobility

BhLGtxUIUAA7U7XOne of the projects I have been involved with recently was a collaboration with the Agents, Interaction, and Complexity group at University of Southampton. The same group who we are also involved with in the Orchid project. This particular project was on Measuring and Predicting Departures from Routine in Human Mobility, building on the PhD work by James McInerney (now at Princeton) and his paper Breaking the Habit.

It is well known that humans generally follow very regular and predicable mobility patterns, both spatially and temporally. Lots of work has looked at exploiting and predicting those patterns but much less so on looking specifically at departures from those regular patterns. What can we learn from those departures from routine? How predictable are they?

Continue reading

Raspberry Pi GPIO Workshop

Update: a writeup of the event by Lorraine is here.

A shot post to say that I am co-organizing a Raspberry Pi workshop with The IET Solent Branch at the University of Southampton on 25 April. For personal reasons I cannot make it myself that day but do go along if you want to learn about how to use the GPIO pins on your Pi board!

Raspberry Pi - 25 April Poster

Btw, if you can’t make it but live in the area, do checkout somakeit.org.uk. Recently launched and we now have a space!

–Dirk

Rewired State Parliament hack weekend 2012

Its 8:44 am and after 5 hours of sleep on my trusty Thermarest I feel quite refreshed, which is more than I can say about the people around me. Some have capitulated and lay scattered around the brightly lit room under their coats in front of their MacBook Air’s. Others are still in exactly the same position I left them 5 hours ago but the intensity has gone and eyes have glazed over. At least one person confirmed the geek stereotype and didn’t manage to hold his beer.

Update: Apparently Parlycloud won a special mention during the judging, thanks!

Continue reading

Coursera-dl: A Coursera download script

I have blogged about coursera.org in the past and as part of signing up to a number of courses I felt the need to easily download the videos, quizzes, notes, etc. locally for later use offline.

I quickly found a project on github (and there are a few) but wasn’t quite happy with the code. I cleaned it up to a relatively sensible state and it now does what I wanted it to do. The main additional features I wanted were: easily download multiple courses, support for quizzes/homeworks, and support for links to extra material (e.g, 3rd party sites, papers, etc).

Just do a “pip install coursera-dl” and then run as follows:

coursera-dl -u myusername -p mypassword -d /my/courses/ algo-2012-001 ml-2012-002

Code is in python and can be found on Github.

Some people have asked if they could donate something. If you wish you can do that here:

Donate Button

Update: if you have a feature request or want to report a bug please use the github issue system

–Dirk

Bridging Python and Matlab: Passing callbacks and optimization

This blog post is about a bridge I created that allows you to pass callbacks to Python functions directly to Matlab code without having to resort to things like sockets or COM.

Background

Designing our UAVs involves a number of different codes that interact & must be coordinated to solve what is essentially a Multidisciplinary Design Optimization (MDO) problem.  One of those codes is written in Python and involves a constrained optimization problem of the following kind:

Find the minimum of a problem specified by

x, b, beq, lb, and ub are vectors, A and Aeq are matrices, c(x) and ceq(x) are functions that return vectors, and f(x) is a function that returns a scalar. f(x), c(x), and ceq(x) can be nonlinear functions.

In our case we have 10 variables (x has length 10), 28 nonlinear constraint functions c(x), 4 nonlinear equality constraint functions ceq(x) and box constraints on most variables.  The objective function f(x) and all constraint functions are pure python functions.

Thus to optimize f(x) you need an optimizer that can handle python functions, and for that there is the excellent openopt library.  This is what I did and after trying out the different NLP solvers I settled on the scipy_slsqp solver for most runs as it seemed to consistently give the best results.

However, as the optimization problem was tweaked and as we did more runs, we did run into cases where it would not solve and the aeronautical engineers I work with were not always happy with its robustness.  Also nobody had ever worked with the openopt library but everybody knew and trusted the various Matlab solvers.  Besides, us being part of the Computational Engineering Design Group, a number of powerful in-house Matlab solvers had been developed and were available to me.  Thus, the least I could do was to try to use those and see if that improved anything.

Python-Matlab bridge

So I started searching the web for tools / libraries that allowed me to bridge python/Matlab, specifically tools that supported passing callbacks between the two languages.  A promising project seemed mlabwrap but while it was great for evaluating Matlab expressions,  it does (at the time of writing) not support callbacks.

In the end I decided to see if I could scratch my own itch.  I really did not want to deal with sockets or any kind of network protocol and I also wanted the solution to be crossplatform, so no COM.  Since both Python and Matlab have well documented interfaces with C/C++ code, that seemed to be the way to go.

So I sat down with the ctypes & mex documentation and with a little help from Mathworks support was able to cobble together a simple example using fminsearch. ¬†By the way, I had never done any Python – C work before, but ctypes is awesome and a joy to use. ¬†The little trick to make it all work is that you just have to treat the function handle as an integer (it’s just a memory address after all). ¬†Then all you are really doing is passing an integer around which is straightforward.

With fminsearch working the goal was to extend to fmincon since that would give us all the basic functionality needed.  This took a bit of debugging and deciphering Matlab crash dumps but I finally got it working.  The final flow of events looks like this:

You start off with writing a matlab file that wraps the optimizer you are interested in (in this case fmincon) and exposes its signature.  This m file forwards all function handle arguments to a mex file which converts the integer to a function pointer and calls it.  This is how we get the callback to a C function.  A simple bridge library is then needed which is loaded by ctypes and converts the arguments into something the Matlab shared library file understands.

From my initial tests I see it is finding the same optima as scipy_slsqp was using openopt. Now I just need to do more extensive testing to see if performance is any more robust. After that I can start looking into the other solvers provided by Matlabs optimization toolboxes as well as our own solvers.

Source code

All the code is available on github. ¬†I hope it is useful to somebody and I’d be very happy to receive patches or improvements. ¬†I guess this may even be of use to the mlabwrap / openopt projects.

–Dirk

Imperial College Robotics Society Talk

I meant to post this some time ago but craziness at work and an overdose of family during the festive period intervened.

Anyway, mid December I did a talk of our UAV work at the Imperial College Robotics society.  I had met Harry Eakins, founder and current president, at the last pyconuk conference in Coventry & he invited me to give a similar talk at Imperial.

As usual, the slides are available on slideshare:

“Click here” to build your UAV

View more presentations from dgorissen

In all I found it a great experience.  I tried to give a broad overview of the UAV related activity at the university as well as focus more on our research around DECODE.  Got a good audience with some great discussion during the reception afterwards.

On arrival I got a tour of their¬†hackerspace¬†and it was great seeing people hacking and soldering away on various projects, ranging from quadcopters to a robotic dustbin. ¬†Thinking back to my university days I always regret not having gotten involved more in such things or set them up if they did not exist. ¬†There are¬†so many cool things¬†you could do and the technology & kit is very accessible. ¬†These days my tinker time is very limited but Im optimistic in that will change ūüôā

–Dirk

“Click here” to design and build your UAV

This blog post summarizes the work I have been doing the past 8 months or so on an automated design system for (civillian) UAVs.  Ive been meaning to do this post for a long time but wanted the tools to be in a beta quality state before writing about them.  In good software engineering tradition there have been a few hiccups and detours along the way but Im happy to say beta stage has been reached.

I will go over the vision behind the system, its architecture, and the various tools I have used to put it together.  For the impatient, this is the idea: we need a system that allows us to rapidly go through this process:

Agile UAV design if you want to call it that. Tools I will talk about include Python, Numpy, RabbitMQ, Redis, Flask, Django, Bootstrap,  Knockout.js, Vagrant and how they link up with application specific codes such as Ansys Fluent, SolidWorks, Vanguard and Anylogic.

This post is an extension of the talk I gave at PyConUK 2011 a couple of months ago and is also what I will cover in my upcoming talk at Imperial College and at the upcoming AIAA conference in Honolulu in April 2012.

Continue reading

RHOK 2011 in Oxford – NGO Clarity

The past weekend I attended my second Random Hacks of Kindness event in the space of a few months.  I went to the Oxford event which was kindly organized by Oxfam and White October.  My initial plan was actually to go to the Belgium event which was organized in my old home town of Antwerp.  But my travel plans ended up being changed so Oxford it was.

The organization was the same as other hackathons I had been to: organizers welcome everybody, idea owners pitch their ideas, chaos, teams are formed.  I quickly settled on the NGO Clarity idea by TechSoup Global Data Acquisitions Manager Dinesh Venkateswaran. With the drive to increasing transparency in the aid sector, there is a lot of data made available on NGOs.  The problem is that the format and quality of the available data varies wildly between and within countries.  Standardization efforts like IATI are trying to change this, but there is still a long way to go.

The goal of the NGO Clarity problem was to setup a system that would analyse the available data on a particular NGO and, depending on a set of rules, output a number of scores indicative of the quality of the provided data (notof the quality of the NGO itself). Such a system would allow a donor or organization like TechSoup quickly evaluate how much effort would be needed to evaluate an NGO.  Are there just a few small details missing or are there large gaping holes/inconsistencies in the data?  Solving this involved a three step process:

  1. Data Aquisition: Dinesh had about 40000 records on Indian NGO’s with him that needed to be loaded and pre-processed
  2. Inference: using heuristics, predefined rules & machine learning to assign scores to each NGO
  3. Presentation: presenting the results in an intuitive manner in the form of a website

While we could not solve every step in detail, my hope was to at least setup an end to end proof of principle.

Our initial team consisted of 4 people, including Dinesh, but unfortunately one soon left to join another project.  After some discussion it was decided that all three steps would be implemented using XPath/XQuery & the XML eXist database.  Since this was the particular specialty of 3rd team member Chris Wallace.  By the first show and tell we had a prototype working, but unfortunately Chris had to leave by then and would not return the next day.  Two others (Russel and Tim) were then kind enough to join me to continue the work.

However, the problem was that none of us was particularly familiar with XQuery. ¬†I setup the eXist database & managed to run Chris’ scripts but we were pretty much stuck trying to extend it in some meaningful manner. ¬†Russel then gave up and left, & Tim had another project to attend to, so it was essentially just me to continue. ¬†After some more poking at the code I also decided to give up as well. Progress was just too slow and as nobody else at the event had any knowledge of XQuery or was willing to help pick it apart I would never get it finished.

Thus, on the train home I decided to start from scratch. ¬†Since I’ve been doing a lot of python lately it was natural to use that. ¬†Though if I were to start this from scratch as a hobby project I would have used Rails or Play, just to get to know those as well. Anyways, I used¬†Django on the backend and jQuery/Bootstrap on the frontend. ¬†About 2 hours later, just when my train arrived I was at feature parity with the xquery stuff. Some more furious hacking the next morning, Ben Foxall helping out with the templates, and about 10 seconds before the presentation was due to start I pushed the demo code to github. ¬†I tried to get a map view to work as wel (geocoding the NGO addresses & plotting them using Google Maps) but couldn’t quite get it to work in time).

Our final presentaion, excellently prepared by Dinesh is up on slideshare:

In sum, it was another great learning experience. ¬†In good hackathon tradition the code is rather hairy so I plan to take some time to clean things up & hopefully improve the rules & structure. ¬†I think its a great idea so I would like to continue working on it as there is much that still needs to be done (see slides). ¬†Unfortunately time is in short supply these days, but let see how it goes… ūüôā

–Dirk