Work in Progress

Teaser update

I haven't posted in a long time and that is partly because I was busy on some stuff I really want to post here.
So this post is a teaser about projects I hope to finish soon.

Sudoku Solver

The first project is a Sudoku Solver. I have liked Sudoku's for a long time and love to philosophize about some interesting features. I know that making a Sudoku Solver is like a first year's Computer Science assignment, but I decided to try and figure it out by myself without any help from other sources.

So far the program is able to solve what you would call Hard Sudoku's, but not the most difficult ones.
Most puzzles only require you to take away the possibilities to find the answers.
For the hardest Sudoku's you have to take more than one thinking step to find more answers. I have plans to improve the algorithm and I believe it will be able to solve any (solvable) Sudoku.

I've made it in Matlab, which is really handy to work with on your own computer, but not so nice to show to other people across the internet. So I still have to do a lot of work to get it online.

Computer Controlled Boat

My first Arduino project was a Led Cube which was a lot of fun, but I was dying for something new and more challenging. When I found a small RC boat at my work I immediately knew what I wanted to do.
I screwed open the remote control and looked where the buttons were giving their signals. When you place a voltage on the right pins you get the same effect as pressing a button. The Arduino is perfect for that.
You can allow the Arduino to send signals to the remote control depending on some input. We could use the keyboard to control the boat for instance. I decided to use Processing, which is very similar to the Arduino language, to communicate with the Arduino via the USB cable.
The whole setup looks this:

On the left we see the Arduino Uno, on the right the opened remote control and to connect the two we see a breadboard in the middle. On the top we see the boat in question upside down so the rotors can turn while testing.

To make this project completely awesome I'll connect a webcam to my laptop to monitor the position and direction of the boat to make it autonomous.
This project is going pretty well so I hope to finish it in a couple of days.

So stay tuned!

Muslim pilgrims per country

I seem to making a lot of maps lately, but it's gives these nice pictures that tell a good story.

Leiden is home to couple of fine museums one of which is Museum Volkenkunde (Museum of Ethnology). At the moment there is an exhibition on the Hajj, the pilgrimage to Mecca called 'Longing for Mecca'. It's on display until March 8th so you have to quick to go and see it.
My girlfriend, Elke Spiessens, did some research for this exhibitions especially tying to find the numbers behind the Hajj. Such as how do people travel to Mecca, or how many stones were used to throw at the Jamarat (stoning of the Devil) and also how many people travel from which country.
That last one struck me as an opportunity to make a nice visualization. These numbers are hard to find exactly and for a lot of countries they couldn't be found so there are some gray spots on the map.

In this map I included a bar plot so you can compare the different countries a bit easier and I thought filling in the numbers with arrows to the countries gives you a bit more feeling than just the colors. Small note, I didn''t have the number of pilgrims that comes from inside of Saudi Arabia. Also most numbers are estimates and based on quotas that are set. So every country can only send a certain amount of pilgrims, which is done for crowd control. The Hajj take place during 5 days in which over 3 million pilgrims visit Mecca and the surrounding sites.

Muslims per country

Looking back at this map I can strike you as odd. The top three countries that send the most pilgrims are Indonesia, Pakistan and India, where the first and third might not fit your typical middle eastern Muslim premise. 

If we look at the map found of Wikipedia showing the percentage of Muslims per country we see the following.

The percentage of Muslims per country

Which is I guess more what you'd expect. Countries in the Middle East and Northern Africa colored dark where a large majority of the people are Muslim (sometimes close to 99%). If you look carefully you can already see that Indonesia has a large Muslim majority  (about 86%). But if we look at the map showing the absolute number of Muslims per country we can understand the map with the number of pilgrims a lot better.

Absolute number of Muslims per country

This map also comes from Wikipedia and can immediately see what's happening. The top three countries Indonesia, Pakistan and India have a lot of citizens and so even if the percentages aren't as high as the Middle Eastern countries it adds up. So it turns out that almost two-thirds of the Muslims live in Asia-Pacific and I think that is a nice eye-opener.

How it's made

I just want to say a few words on how this map came together. As I mentioned in previous posts a great inspirational source is Visualize This by Nathan Yau and making this map was helped a lot by the examples found in that book. The data I had was run through a Python script to create the CSS styling needed to color in the SVG map. The bar plot was made in R, exported as a PDF, imported in Inkscape, styled to fit the map and then added. All the numbers and arrows were put in by hand in Inkscape.

Since this post is more focused on the story than on the technique I hope you find it informative. If you like to learn more about the Hajj and the people that take this pilgrimage go see that exposition before it closes.

How close are you to a school building?

In Leiden, The Netherlands there was quite a stir when a convicted pedophile was placed in a new house after being released. This left people in the city and especially the neighborhood angry and concerned. One of the points the mayor of Leiden made was that everybody needs to live somewhere. People who objected said out that there are schools nearby the house that was appointed and that of course made it a bad choice.

These arguments made me think. Is it possible to find a house that is not near a school?
My initial guess was that it was nearly impossible.

The data

So, to lighter subjects. Let's make a map! I first looked for a list of all the school in Leiden. That includes higher education. I must say that afterwards I realized I probably missed some school buildings. I didn't check if a certain primary or secondary school had more then one location. I also forgot some of the higher education facilities. And lastly I totally ignored kindergartens and such, which in my guess adds 30+ more locations.

The list I found for schools had addresses and postal codes included, as well as the type of the school. It took some time to get everything in a nice spreadsheet, but it was ok. I used Google's mapsengine to put all these locations on a map. There are some simple tutorials on the basics of Mapsengine, so within a few minutes I had a nice looking and useful map. I made a screenshot as you can see below or you can follow this link to play with interactively.

Locations of schools in Leiden

If you follow the link you'l be able to click on every location and see which school it is and it's properties.

You can also put your mouse over of one of the schoolmtypes in the list on the left and the school that belong to that type are highlighted. Pretty neat right. You can style this in a couple of ways and share it across the web.

Mapsengine is a really easy tool to use but in my opinion it's possibilities are restricted. For instance the border of the municipality had to be drawn in by hand. Since I only had to do this once this was no big deal. But Google Maps draws this shape if you search for Leiden, why isn't that feature included here?

Circles

When working on this is was thinking of a site that used Google Maps to show the impact of different atomic bombs on your favorite location. It draws circles around a position to show which areas are destroyed by the fire, which by the air blast, etc etc. Seems simple right, draw a circle around a point on the map. Sadly Mapsengine doesn't have this feature. What I wanted to do was draw a circle around every school with a radius of let's say 500 meters. I mean, that's close by right? Some browsing led me to a site called Freemapstools. You can upload a list of GPS coordinates plus the radii of the circles you want to draw and it makes a map with said circles. I did have to get all the GPS coordinates first. For that I used the website GPSCoordinaat, which only works for dutch addresses I'm afraid. I was just to lazy to find a different way to transform that data so I just filled in every address by hand. Sometimes doing a monotonous job is faster than trying to automate it.

Plugging all the coordinates in Freemapstools gave me the following map.

Circles of 500 m around school locations

Freemapstools allows you to export this map to a .kml file which you can import in Google Earth. (please comment if this doesn't work) That should look something like the image below.

Google Earth version

This allows to some zoom in nicely to check out those few areas that are not covered by a circles. Some of the empty spots are industrial areas or sport facilities and a large part is the city's center. Remember that I forgot a couple of locations and I didn't include kindergartens so a proper map would be probably be filled more than the above maps.

Conclusions

So what can we conclude from all this? 

1) Making a map is still a lot of work and not fully satisfactory

I really wanted to have both maps in one. The locations with the information on every school and the circles around them. And it would have been even better if those radii were adjustable so it would be more insightful. The radius of 500 meters was just a number I picked that sounded nice. A different radius like 200 meters or one kilometer would create a different story.

2) You're always close to a school building in Leiden

Leiden has a lot of people in a small area and a lot of schools, you can't really change that.

Map of Belgian municipalities and JSFiddle

In previous posts in showed the work I did on maps of The Netherlands. After that I thought it would be nice to see if I could do the same for Belgium. A benefit of looking at Belgian municipality borders don't change as much as the Dutch ones do.
As ever Wikipedia was a good source for a SVG version of the map I was looking for.

Map of Belgium with all the municipalities

There were only two problems. In the most recent maps of The Netherlands every municipality was a path and had it's name in the Id tag. You've guessed it, the Belgian map didn't have this. All the borders between two areas were a separate line. One municipality could be defined by maybe 5 lines but not one single area. So I needed some way turning all these lines into closed paths and putting the right name in the right path.

Inkscape SVG editing

I must add that Belgium has a very complex political system and maybe that is why they have over 400 municipalities.
To go from the border lines to paths I used a simple bucket fill in Inkscape. I think you lose some resolution in this process but for what I intend to do it's just important to get all the areas in the right place.
I found out that Inkscape has an Object Properties window (shift+ctrl+o if you're wondering) were you can change i.e. the Id and Title properties of an object in your SVG file. I had to use several .jpg maps which had the names of the municipalities on them and overlaid them with the empty SVG map.
This process cost me a couple of hours but I'm pleased with the result.

JSFiddle

It's possible to put an SVG directly into a blog. If you go to the HMTL mode you can just paste the SVG code in your post and it should show in the final version. But if it's too big it doesn't seem to work. So to show the map I made I had to try something else that didn't involve making my own website. I chose JSFiddle, a nice website to allows to create an account and to try out HTML, Javascript, etc and to share it with other developers.
The map, as is, can be found here. In this version of the map every municipality is one path, where the Id name and the Title are filled in. So if you hover with your mouse over an area you can see it's name (Title).

Future Work

If you would want to color every path individually to make a choropleth you still need to do some work and I intend to do that.
One thing I would like to add to the map is the different regions that exist, the governments, the districts, etc.
I also need some data to visualize. The Belgian government has some sites where they share data, but for now I didn't find them that insightful.
When I've made some examples I hope to post them soon.

Belgian Map

This isn't a long post, it's more of a test and a preview.
As I worked the maps of The Netherlands I thought of doing similar maps of Belgium.

Empty map of Belgium

Wikipedia was a helpful source of maps again, but with a downside. Where the map of The Netherlands had information about the region inside the SVG file the Belgian maps have none.
So if you want to assign some color to a certain area you have to make a translation between i.e. 'Luik' and 'path2275'. But since there is no such list available I have nothing to work with.
The easiest work around is to open a map in Inkscape and manuallly change the 'Id' of every path. All I needed was a map (a regular .png file) with the names of the regions (as below) and the empty map (as above).

Map of Belgium with names of provinces

Since there are only 10 provinces and Brussels this wasn't a lot of work. Now the file is almost ready to be used as I described in my post My first choropleth map. You just have to remove the 'fill' attribute and create a list with the fill per province. Either in the file itself or in an external css file.

Future plans involve doing the same for all the municipalities (589 in total) and finding data to create some visualisations. 

Sharing is nice

Lastly I want to add the SVG file I edited mainly to test if readers can download files from posts and also to share the work. If you've tried downloading it and it doesn't work, please let me know. If it worked and you liked it, I would also like to know :)

Link to the file.

My second choropleth map

Municipalities

In my last blog entry I wrote my first attempt at coloring areas in a map based on values associated with that area. In that case it was the number of people living in a province of the Netherlands. I thought of expanding that by doing the same for municipalities in the Netherlands. Sound like the same routine right, but just a bit more numbers. Well you've guessed it, it wasn't.

The easy part

First I looked at all the available maps on Wikipedia. There's at lot to choose from so I downloaded every empty map just to be sure. The problem is that not all maps are scalable vector graphics. The most recent maps are, and they have the nice feature that in the paths that describe the boundaries the name en area code are added. For example the first one is Appingedam with area code GM0003. Since these are official names and codes I was feeling confident that I would succeed to connect the right paths to the data I wanted to use.

A bit harder

I had decided to color the area of every municipality based on the number of inhabitants. I guessed that this data was really easy to find. And sure the most recent numbers you can the internet quickly enough. But I was thinking of comparing different years and finding that data needed some digging in the CBS files. I found out that the data was available but not in handy spreadsheets for me to download. What I did download was a pdf containing demographic information per municipality. There's a file like this for every year dating back to 2000 so that seemed like a nice start.
In this file information is grouped per province. So I took the time to copy-paste the pages I needed in a plain text file to use later on. The first line in that file looks like this:

0003 Appingedam 12 114 2 598 1 170 1 463 1 813 2 663 1 682 725 19,9

The first number is the area code as mentioned above (hurray), the second is the name (yeah) and what follows is lot's of numbers. In the original file a space is used to divide groups of three digits to improve readability. The first two numbers after the name therefore together make up the total number of inhabitants, in this case 12114.

Regular expressions

Since I felt this should be a learning experience I decided this would be a great opportunity to get my hands dirty with some regular expression magic in R. I finally came up with the following expression that works in this particular case:

regexp <- digit:="" i="">

Short explanation: :digit: makes it look for a digit and the brackets {} tells the number of digits. When it says {1,3} it's happy if it there are 1 to 3 digits. Somewhere in the middle there's a (.+) that is just looking for anything. In each line of the data file R tries to extract a piece that fits the description in the regular expression and puts that in a new list. The first element would be:

0003 Appingedam 12 114

Luckily this worked for the entire list I had with one exception. I assumed beforehand that every municipality had at least 1,000 inhabitants but it turned out that Schiermonnikoog only has 948. That mistake was easily found in the end when I was wondering why a small island, which in my opinion was pretty peaceful, seemed to have at least 800,000 inhabitants.
Breaking apart the string that remained wasn't hard after that. I just had to take into account that some names consist of more than one word, i.e. Kapelle aan den IJssel. The easiest solution was just taking off the last two elements in the string, since those are the two numbers that together make up the number of inhabitants.
Finally exporting the now nice looking data in a csv file and I could start thinking about using it to color the map.

IndentationError

Throughout this process I was following the example by Nathan Yau on making a choropleth map of U.S. counties. He was using Python with the BeautifulSoup library. Now I have my map finished I must I like it a lot. But times were different. Since this was my first real Python experience it took a lot of time understanding all the small things that can make a python script crash and burn (like a misplaced space).
In the end I had my data in a dictionary ready to be accessed when a certain path needed to be colored and also to find those area codes inside the svg map when needed. The end result (converted to png) looks like this.

There are some problems with this map. There's no legend or title for starters, but there are also some gray areas. Municipalities merge sometimes, and I had used the data from 2009 on a map of 2012. So that was just a dumb mistake. The third problem is the colors. I hadn't noticed the gray areas on first glance since I'm colorblind. The gray areas somehow just seem to blend in with the other colors, so I really want to find something in the future to make sure there's a better distinction between colored and uncolored areas.

Touching up

Just to make the final image look nicer I added a legend and a title in Inkscape before I did a run of the python script. I'm pretty pleased with the end result as it is, even with the before mentioned problems. Check it out below.
What I really hope to look into is the following. Now the svg file that first plain had gray areas now has colored areas so the entire file is altered. But if you would like to change the image you have to run the python script again. If you're thinking on using something like this on a website with interactivity that doesn't sound like it would work properly. You would want a svg image where the fill of the path can be changed whenever you like. So, back to the drawing board.

My first choropleth map

I just want to show some SVG features. I made a map of the Netherlands where the provinces are colored determined by the number of inhabitants. I've been reading Visualize This by Nathan Yau (from FlowingData) and looking at the examples about maps I figured I should try it out myself.
I wrote a small R script to go from a list of inhabitants per province to a css list of gray tints.
A SVG map of the Netherlands was easily found on Wikipedia. This is just a great source of empty maps in vector format, where paths and groups already have id labels which you can use. In Inkscape I edited the paths so that there was only one path per province.

Since I know little to nothing about HTML, CSS, SVG coding (yet!), it took some time to find a way to use the list I made of gray tints with the fill features for every path without filling it in by hand. In the end put a group around every province path and added a 'class' with the name of the province, i.e. class="Drenthe".
To change the fill color I put the following in the style section of the SVG:

.Drenthe{fill:#202020;}

I'll put the SVG below so you can check out the details by downloading it and opening it in your own favorite text editor (notepad++)

So looking back on this small project I guess you always need to put some effort into preparing your data (getting grays in this case), but also into preparing the map. The map already looked good, but beneath the hood I had to do some work. Now it's possible to put in a new list of colors and your new map is ready. Since I have the R script to produce such a list it would only take me a couple of minutes to get a brand new map.
I added some hover functionality to test that and it works, but this is of course from a nice looking map. There's no title and no legend, but those are things to look for in the near future. P.S.: I forgot that older versions of Internet Explorer don't support SVG, so if there's no map below that's probably it.