Lightweight, Cross-platform, Real-time Browser-Browser Communications

During a monthly meeting to discuss cutting edge technologies here at the Biodiversity Informatics Group at the Marine Biological Laboratory, I demonstrated a technique to update distributed browsers in the face of collaborative classification (i.e. tree) editing. In essence, if there are 2+ people asynchronously (i.e. via AJAX calls) updating content on a web page, there is potential for everyone to get horribly out of sync with one another. Imagine for example a chat window on a web page that does not update on everyone's web page in real time....wouldn't make for a particularly pleasant or useful experience for anyone. The same lousy experience was true in the LifeDesks tree editor when 2+ people were simultaneously updating the same classification. Person A might delete or move a node and person B, C, D, ... etc. are none the wiser and might later perform an action on that node (or its children) whereas the database no longer reflects what they see in their browser screen.



To work around the possibility that everyone editing can get horribly out of sync with one another, I implemented a polling mechanism to grab recent adjustments to data every 5 seconds. If you happen to be looking at a portion of the tree that someone else has just deleted or moved elsewhere in the tree, relevant nodes within the tree will now automagically refresh to reflect actions that someone else just did...nodes will flash red then disappear, nodes will flash green then appear, etc. There is also a scrolling activity monitor at the bottom of the screen. To be sure, this isn't a particularly robust mechanism because there is constant polling. Enter web sockets...


Ryan Schenk who attended this informal demonstration alerted me to Socket IO. I knew of it, but never paid much attention. However, after having poked around a little bit with the examples provided, I am convinced this is the way I should have designed real-time classification tree updates in the face of 2+ simultaneous user actions. The lightweight technique will prove useful for any client-client communications (e.g. real time chat). Plus, it has the excellent benefit of cross-browser, cross-platform capabilities with very little server strain. A database need only be hit once when person A exerts an action and the data propagates to all other users. Very cool.

MapServer, MapScript, MacPorts

For anyone wishing to get into MapServer and serve shapefiles via PHP and also use a Mac with MacPorts for local development, here is how to compile. I discovered the hard way that the MacPorts port for MapServer is horribly dated and DOES NOT include PHP-MapScript. Compile instructions below assume you already have the php5 MacPort.

Install some dependencies if you haven't already got them:

sudo port install php5-gd
sudo port install xpm
sudo port install proj
sudo port install geos
sudo port install gdal

1. Download latest MapServer tarball, http://mapserver.org/download.html (e.g. at time of writing http://download.osgeo.org/mapserver/mapserver-5.6.5.tar.gz)
2. Extract and cd into folder
3. Execute from command line:

$ ./configure \
--prefix=/usr \
--with-agg \
--with-proj=/opt/local \
--with-geos=/opt/local/bin/geos-config \
--with-gdal=/opt/local/bin/gdal-config \
--with-threads \
--with-ogr \
--without-tiff \
--with-freetype=/opt/local \
--with-xpm=/opt/local \
--with-png=/opt/local \
--with-jpeg=/opt/local \
--with-gd=/opt/local \
--with-wfs \
--with-wcs \
--with-wmsclient \
--with-wfsclient \
--with-sos \
--with-fribidi-config \
--with-experimental-png \
--with-php=/opt/local

4. Execute from command line: $ make
5. Verify that mapserv is working by executing ./mapserv -v
6. Find php_mapscript.so in mapscripts/php3 and move to PHP extensions directory (usually /opt/local/lib/php/extensions/no-debug-non-zts-20090626/ for MacPorts). You may also need to add php_mapscript.so to your php.ini.
7. Move mapserv into cgi-bin folder for web server and give permission to execute if desire using it directly (optional)

If MacPorts's GDAL were similarly updated to v. 1.7.3, you could use GeoRSS data just as you would use shapefiles. But, alas, at the time of writing, the version in MacPorts is v. 1.6.2.

While we're on the mapping kick, here is a very excellent source of shapefiles: http://www.naturalearthdata.com/

...and a bit of PHP code to consume GeoRSS using the Magpie RSS library. The author uses some deprecated PHP functions in places, but it is nonetheless quite useful.

Reference Parser Revived

Many moons ago, I once developed a tool that does real time discovery of scientific references using a combination of the aged (though still very useful) ParaTools and CrossRef's OpenURL service. With the demise of my server, this bit of code was lost. I just revived the code and functionality and provide it here for anyone else to take it and refine it UPDATE: parsing is now executed with a Ruby gem: http://refparser.shorthouse.net/. This location is not likely to persist so get it while you can. To get a sense of what it does, here are some verbatim references. Click the magnifying glass after each reference to experience the magic. Cross-domain AJAX requests are circumvented by using jQuery's clever JSONP handling.

Bell, C. D., & Patterson R. W. (2000). Molecular phylogeny and biogeography of Linanthus (Polemoniaceae). American Journal of Botany. 87, 1857-1870.

Epling, C., & Dobzhansky T. (1942). Genetics of natural populations. VI. Microgeographic races in Linanthus parryae. Genetics. 27, 317-332.

Epling, C., Lewis H., & Ball F. M. (1960). The Breeding Group and Seed Storage: A Study in Population Dynamics. Evolution. 14, 238-255.

Similarly, this can be done with an input box. Paste a reference and press enter:

Authentication Made Easy

I am swamped by the number of user names and passwords I have to remember and, quite frankly, if a new resource I stumble upon requires me to remember yet another account for me to access or do something I need, it's a deterrent and I'll go elsewhere. While developing features for SimpleMappr, it occurred to me that users probably would like to save a template of a naked map and then populate it with various bits of data at various times. In other words, it would be handy to just draw-up a template and use it whenever creating something new. Rather than making yet another user account system (ugh!) for this map template saving tool, I made use of Janrain's (formerly RPX) OpenID system. In less than an hour, I made a 2-click user authentication system for users. While Janrain is a for-profit company, it's only a matter of time for an open-source equivalent at which time I can probably just switch and not have to adjust the database schema or much of my code.

SimpleMappr API

There are plenty of resources to make pushpin maps, but none that I know of have a Microsoft Excel add-on to make use of these. My ultimate goal is to make one as part of SimpleMappr to help streamline map creation for assembly into manuscripts. To get a little closer to this vision, I spent a half-hour making a RESTful API and the documentation may be found here: http://www.simplemappr.net/#map-api.

For example, this URL: http://www.simplemappr.net/api/?file=http://www.simplemappr.net/public/files/demo.txt&shape=square&size=10&color=255,0,0&width=400&bbox=-130,40,-60,50&layers=lakes,stateprovinces&projection=esri:102009

Gives you this:

Mapping Revival

One of the casualties of the death of the Nearctic Spider Database was a largely neglected, simple mapping application that permitted copy/paste of collection coordinates. The output was a b&w line map with contoured dots, mostly suitable for insertion in manuscripts. Sadly deficient was the ability to have many layers, each with different pushpin style or to crop, zoom, pan or change the projection. Unbeknown to me, folks were actually using this thing. I actually made a much better application for the AMNH to help produce outputs for their PBI grant holders. So, with Norm Platnick's permission, I re-purposed some of the code.

Here be SimpleMappr, http://www.simplemappr.net.



There are bound to be bugs or hiccups because it's never been fully tested under load, but I throw it out here for feedback and feature requests. Please let me know what you were doing when and if you witness odd behaviour; this is a very dynamic environment. Yes, there are similar sorts of applications out there in the wilds but none to my knowledge permit this sort of facile "copy/paste/tweak/export" as fast as this one can.

Nearctic Spider Database Dead

With great sadness, I will no longer be serving the Nearctic Spider Database unless something remarkable happens.

On March 17, 2010, the power supply sparked in my server, shorted out the motherboard and as a consequence, the hard drives seized up. While I of course have back-ups, unbeknown to me the incremental drive image for the applications portion of the server was corrupt. The latest working drive image was January 2007 - hardly useful to rebuild the server. This means I have to reconstruct the server from bare metal, which would be a significant financial hit and a significant consumption of time away from family.

The website currently serves a flat html page where one may download the code and data until March 31, 2010 at which time it will evaporate.

I estimate it would take a solid week to re-install and iron-out the kinks. But, if it takes that long, surely it would be better to have a fool-proof system. And, in particular, one NOT dependent on Microsoft software.