Realtime Web

I started work on a whimsical presentation I will soon give to the Biodiversity Informatics Group at the Marine Biological Laboratory about the Realtime Web and came-up with the following kooky slide. Felt the urge to share.

Amazing Web Site Optimizations: Priceless

Quite literally, priceless. As in costs nothing.

I was obsessed with web site optimization these past few weeks, trying to trim off every bit of fat from page render times. As we all know, if a page takes longer than approx. 3-4 seconds to render, then you can expect to lose your audience. Even though expectations for speed vary depending on the end-user's geographic location, having a website that can be equally fast for a user in Beijing is just as important as the experience for a user in California. As might be expected, server hardware typically isn't the bottleneck. Another way of looking at this is to recognize that remarkable boosts in performance can be had on crap hardware. So, this post presents the tools I used to measure web site performance and describes the simple techniques I employed to trim the excess fat.

My drug of choice to measure the effect of every little (or major) tweak has been WebPagetest, a truly invaluable service because I can quickly see where in the world and why my web page suffered. Knowing that it took 'x' ms to download and render a JavaScript file or 'y' ms to do the same for a css file meant I could see with precision what a bit of js or css cleansing does to a user's perception of my web site. I also used Firebug and Yahoo's YSlow, both as FireFox plug-ins. Google Chrome also has a Page Speed extension that I used to produce a few optimized versions of graphics files.

Some tricks I employed to great effect, in order from most to least important:

1. Make css sprites. The easiest tool I found was the CSS Sprite Generator. Upload a zipped folder of icons and it spits out a download and a css file. Could it be any easier? Making a css sprite eliminates a ton of unnecessary HTTP requests and is by far the most important technique to slash load times.
2. Minify JavaScript and css. For the longest time, I was using the facile JavaScript Compressor, but the cut/paste workflow became too much of a pain. So, I elected to use some server-side code to do the same: jsmin-php and CssMin. When my page is first rendered, the composite js and css files are made in memory then saved to disk. Upon re-rendering (by anyone), the minified versions are served. Here's the PHP class I wrote that does this for me. Whenever I deploy new code, the cached files are deleted then recreated with a new MD5 hash as file titles.
3. Properly configured web server. This is especially important for a user's second, third+ visit. You'd be crazy not to take advantage of the fact that a client's browser can cache! I use Apache and here's what I have:
   
   <Directory "/var/www/SimpleMappr">
   Options -Indexes +FollowSymlinks +ExecCGI
   AllowOverride None
   Order allow,deny
   Allow from all
   DirectoryIndex index.php
   FileETag MTime Size
   <IfModule mod_expires.c>
   <FilesMatch "\.(jpe?g|png|gif|js|css|ico|php|htm|html)$">
   ExpiresActive On
   ExpiresDefault "access plus 1 week"
   </FilesMatch>
   </IfModule>
   </Directory>
   
   Notice that I use the mod_expires module. I also set the FileETag to MTime Size, though this was marginally effective.
4. Include ALL JavaScript files just before the closing body tag. This boosts the potential for parallelism and the page can begin rendering before all the JavaScript has finished downloading.
   
5. Serve JavaScript libraries from a Content Delivery Network (CDN). I use jQuery and serve it from Google. Be weary that on average, it is best to ONLY have 4 external sites from which content will be drawn. This includes static content servers that might be a subdomain associated with your web site. Beyond 3 external domains or subdomains, DNS look-up times outweigh the benefit of parallelism, especially for aged versions of Internet Explorer. Modern browsers are capable of more simultaneous connections, but we cannot (yet) ignore IE. I once served jQueryUI via the Google CDN, but because this was yet another HTTP request, it was slower than had I served it from my own server. So, I now pull jQuery from the Google CDN and I include jQueryUI with my own JavaScript in a single minified file from from my server.
6. Use a Content Delivery Network. I use CloudFlare because it's free, was configured in 5 minutes and within a day, there was noticeable global improvement in web page speed as measured via WebPagetest. Because I regularly push new code, I use the CloudFlare API to flush their caches whenever I deploy. However, this is largely unnecessary because they do not cache HTML and as mentioned earlier, I make an MD5 hash as my js and css file titles.
   

So there you have it, I was able to trim 4-6 seconds from a very JavaScript-heavy web site. And, web page re-render speed is typically sub-second from most parts of the world. Because much of the content is proxied through CloudFlare, my little server barely breaks a sweat.

Did I mention that none of the above cost me anything?

SimpleMappr Embedded

I never had high hopes for SimpleMappr. There are plenty of desktop applications to produce publication-quality point maps. But it turns out, users find these hard to use or are too rich for their pocket books. As a result, my little toy and its API are getting a fair amount of use. I find this greatly encouraging so I occasionally clean-up the code and add a few logical, unobtrusive options.

A number of users appear to want outputs for copy-paste on web pages and not copy-paste into manuscripts, so I just wrote an extension to permit embedding.

Here's one such example using the URL
http://www.simplemappr.net/?map=643&width=500&height=250:



Happy mapping...

Lightweight, Cross-platform, Real-time Browser-Browser Communications

During a monthly meeting to discuss cutting edge technologies here at the Biodiversity Informatics Group at the Marine Biological Laboratory, I demonstrated a technique to update distributed browsers in the face of collaborative classification (i.e. tree) editing. In essence, if there are 2+ people asynchronously (i.e. via AJAX calls) updating content on a web page, there is potential for everyone to get horribly out of sync with one another. Imagine for example a chat window on a web page that does not update on everyone's web page in real time....wouldn't make for a particularly pleasant or useful experience for anyone. The same lousy experience was true in the LifeDesks tree editor when 2+ people were simultaneously updating the same classification. Person A might delete or move a node and person B, C, D, ... etc. are none the wiser and might later perform an action on that node (or its children) whereas the database no longer reflects what they see in their browser screen.



To work around the possibility that everyone editing can get horribly out of sync with one another, I implemented a polling mechanism to grab recent adjustments to data every 5 seconds. If you happen to be looking at a portion of the tree that someone else has just deleted or moved elsewhere in the tree, relevant nodes within the tree will now automagically refresh to reflect actions that someone else just did...nodes will flash red then disappear, nodes will flash green then appear, etc. There is also a scrolling activity monitor at the bottom of the screen. To be sure, this isn't a particularly robust mechanism because there is constant polling. Enter web sockets...


Ryan Schenk who attended this informal demonstration alerted me to Socket IO. I knew of it, but never paid much attention. However, after having poked around a little bit with the examples provided, I am convinced this is the way I should have designed real-time classification tree updates in the face of 2+ simultaneous user actions. The lightweight technique will prove useful for any client-client communications (e.g. real time chat). Plus, it has the excellent benefit of cross-browser, cross-platform capabilities with very little server strain. A database need only be hit once when person A exerts an action and the data propagates to all other users. Very cool.

MapServer, MapScript, MacPorts

For anyone wishing to get into MapServer and serve shapefiles via PHP and also use a Mac with MacPorts for local development, here is how to compile. I discovered the hard way that the MacPorts port for MapServer is horribly dated and DOES NOT include PHP-MapScript. Compile instructions below assume you already have the php5 MacPort.

Install some dependencies if you haven't already got them:

sudo port install php5-gd
sudo port install xpm
sudo port install proj
sudo port install geos
sudo port install gdal

1. Download latest MapServer tarball, http://mapserver.org/download.html (e.g. at time of writing http://download.osgeo.org/mapserver/mapserver-5.6.5.tar.gz)
2. Extract and cd into folder
3. Execute from command line:

$ ./configure \
--prefix=/usr \
--with-agg \
--with-proj=/opt/local \
--with-geos=/opt/local/bin/geos-config \
--with-gdal=/opt/local/bin/gdal-config \
--with-threads \
--with-ogr \
--without-tiff \
--with-freetype=/opt/local \
--with-xpm=/opt/local \
--with-png=/opt/local \
--with-jpeg=/opt/local \
--with-gd=/opt/local \
--with-wfs \
--with-wcs \
--with-wmsclient \
--with-wfsclient \
--with-sos \
--with-fribidi-config \
--with-experimental-png \
--with-php=/opt/local

4. Execute from command line: $ make
5. Verify that mapserv is working by executing ./mapserv -v
6. Find php_mapscript.so in mapscripts/php3 and move to PHP extensions directory (usually /opt/local/lib/php/extensions/no-debug-non-zts-20090626/ for MacPorts). You may also need to add php_mapscript.so to your php.ini.
7. Move mapserv into cgi-bin folder for web server and give permission to execute if desire using it directly (optional)

If MacPorts's GDAL were similarly updated to v. 1.7.3, you could use GeoRSS data just as you would use shapefiles. But, alas, at the time of writing, the version in MacPorts is v. 1.6.2.

While we're on the mapping kick, here is a very excellent source of shapefiles: http://www.naturalearthdata.com/

...and a bit of PHP code to consume GeoRSS using the Magpie RSS library. The author uses some deprecated PHP functions in places, but it is nonetheless quite useful.

Reference Parser Revived

Many moons ago, I once developed a tool that does real time discovery of scientific references using a combination of the aged (though still very useful) ParaTools and CrossRef's OpenURL service. With the demise of my server, this bit of code was lost. I just revived the code and functionality and provide it here for anyone else to take it and refine it UPDATE: parsing is now executed with a Ruby gem: http://refparser.shorthouse.net/. This location is not likely to persist so get it while you can. To get a sense of what it does, here are some verbatim references. Click the magnifying glass after each reference to experience the magic. Cross-domain AJAX requests are circumvented by using jQuery's clever JSONP handling.

Bell, C. D., & Patterson R. W. (2000). Molecular phylogeny and biogeography of Linanthus (Polemoniaceae). American Journal of Botany. 87, 1857-1870.

Epling, C., & Dobzhansky T. (1942). Genetics of natural populations. VI. Microgeographic races in Linanthus parryae. Genetics. 27, 317-332.

Epling, C., Lewis H., & Ball F. M. (1960). The Breeding Group and Seed Storage: A Study in Population Dynamics. Evolution. 14, 238-255.

Similarly, this can be done with an input box. Paste a reference and press enter:

Authentication Made Easy

I am swamped by the number of user names and passwords I have to remember and, quite frankly, if a new resource I stumble upon requires me to remember yet another account for me to access or do something I need, it's a deterrent and I'll go elsewhere. While developing features for SimpleMappr, it occurred to me that users probably would like to save a template of a naked map and then populate it with various bits of data at various times. In other words, it would be handy to just draw-up a template and use it whenever creating something new. Rather than making yet another user account system (ugh!) for this map template saving tool, I made use of Janrain's (formerly RPX) OpenID system. In less than an hour, I made a 2-click user authentication system for users. While Janrain is a for-profit company, it's only a matter of time for an open-source equivalent at which time I can probably just switch and not have to adjust the database schema or much of my code.