Monday, July 1, 2013

NameSpotter: Experiences Making a Google Chrome Extension

If you believe various browser penetration statistics, Google Chrome is more popular than any other browser in use today. And, with the imminent demise of the popular iGoogle homepage later this year, my suspicion is that Chrome apps and extensions will grow in popularity...something is no doubt in the works in Mountain View. Many of these apps and extensions are freely available on the Chrome Web Store. Some time ago I decided to poke around and learn what it takes to develop a Google Chrome extension. I was pleasantly surprised at how easy it was. It's a good time to finally write about my experience, especially since I spent a few hours yesterday renewing an extension I designed called NameSpotter, whose code is on GitHub. The first part of this post is a description of what my NameSpotter extension does and the second part is cursory background to Chrome Extension authoring.

What Does the NameSpotter Extension Do?

The NameSpotter Chrome extension puts a Morpho butterfly icon  in your browser toolbar. Click it and its wings start to flap while the content of the current web page under view is sent to the Global Names Recognition and Discovery web service where two scientific name-finding engines, TaxonFinder and NetiNeti take action. A list of scientific names returns and these are highlighted on the page. Run your cursor over a name and a tooltip draws-up content about that name from the Encyclopedia of Life. There's a resizable navigation panel that appears at the bottom of the page that allows you to jump your position on the page from name to name or to copy all or some of the found names into your clipboard for pasting elsewhere. Many of these features are customizable from a settings area in the navigation panel. For example, if you prefer not to have tooltips, you can turn those off.

The interface is in English or French, depending on your system settings. And, only common names (when known by EOL) in the language of your system settings are shown. If EOL had an API that accepted locale as a parameter, I'd use that too. As it stands, if your system settings are in French, you're going to get English content in your tooltips.

If you happend to be viewing a PDF while in Chrome and click the Morpho icon in your browser toolbar, you get a panel at the bottom of the page as usual but you don't get the tooltips. PDFs (sadly) are not HTML so there's nothing I can do to manipulate the static content you're reading.

There's quite a bit of action in this extension with many messages that get passed around from server to your browser and within the extension itself. I also made use of two very excellent jQuery plugins called Highlight and Tooltipster. The most difficult aspect to handle was making sure the extension performs well, especially when there might be hundreds if not thousands of names on the page. This is where a little knowledge about the performance of jQuery selectors comes in handy.

What's Next?

There are plenty of other aspects to this extension that could be explored such as:
  • Auto-indexing URLs and scientific names: Any click of the Morpho that results in found names could send the URL in the browser bar and the names found to an index without the user knowing. This would be equivalent to crowd-sourced web spidering. An aggregator of content like EOL might be very interested to receive URL + name combinations to make an auto-generating outlinks section.
  • Other sources of content in tooltips: I designed the tooltip to be (relatively) flexible. If there are other resources that accept a scientific name as a query parameter in a JSON-based RESTful API, I could wire-up its responses. If you are more interested in nomenclatural acts, I could for example use ZooBanks APIs. Or, I could send the namestring to CrossRef's API and pull back some recent publications. There's really no limit to sources of data. What's limiting is my appreciation what's useful in this framework.
  • Sending annotations to the host: This one's a bit half-baked, but why can't a Google Chrome extension be used to push annotations, questions, comments to the host website? You'd need a bit of OAuth and something on the web page to inform the extension that the host is willing to accept annotations and where to send them. Something like webhooks come to mind.
Do you have other suggestions?

How Do You Make a Chrome Extension?

Google Chrome extensions are remarkably simple, based solely on HTML, css and JavaScript, the basic tools of web page development. In contrast, FireFox and Internet Explorer extensions are horribly complex and their documentation for first-timers is equally terrible. The documentation for Chrome extensions is wonderful with plenty of tutorials and free samples. Development is made especially easy because you can load an "unpacked" extension, tweak it, reload it, and iterate with a few clicks from your Chrome extensions page while in developer mode.

Chrome extensions have three basic parts: 1) Metadata file, 2) Content scripts and, 3) Background pages/Event scripts and a very important construct: Passing Messages.

Parts of a Chrome Extension

Metadata File

The metadata file is a static JSON document called the Manifest and it contains basic information about the extension such as a title, description, default locale (language) as well as more complex concepts such as permissions and the local JavaScript and css files the extension will use. As I learned the hard way, you cannot put bits in your manifest (eg. configuration variables) that Google doesn't expect to see. So, if you have a need for configuration variables as I did, you have to do a bit of AJAX to grab the contents of your static file:

nsbg.loadConfig = function() {
  var self = this,
      url = chrome.extension.getURL('/config.json');

    type : "GET",
    async : false,
    url : url,
    success : function(data) {
      self.config = $.parseJSON(data);

Content Script(s)

Content Scripts are the JavaScript and css files that you want injected into the web page. They run in the context of web pages, but are encapsulated in their own space. You cannot execute JavaScript functions that might already be declared in the source of a web page. Besides, how would an extension know that such a function could be executed? Nonetheless, if you need jQuery or any other library to write your content scripts, drop 'em in a folder in your extension and declare 'em in your manifest. It is that simple. Content scripts do however have access to the DOM of web pages. So, you can modify links, access the pictures, or any other content on the web page via your content script.

Background Page / Event Script(s)

Background pages do as their name suggests - they run in the background and don't require user interaction to do so.  Event scripts are similar to background pages but are more friendly toward system resources because they can free memory when not needed. Why do you need background or event scripts? A good candidate for a background page is material for a user interface. Another important reason for background pages or event scripts is that these have capabilities that content scripts to do not, eg. access to the context menu or bookmarks. You don't always need a content script, but you always need a background page or event script. If you do have a content script, a good rule of thumb is to keep it mean and lean and dump the heavy lifting into a background or event script.

Passing Messages

This is the most difficult part of a Chrome extension, but powerful once you understand why it's needed. Background scripts have access to system-like functions that content scripts do not and content scripts respond to user interaction whereas background scripts (mostly) cannot. You bridge the two worlds by passing messages.

Here's a method in a content script that broadcasts a message:

chrome.extension.sendMessage(/* JSON message */, function(response) {
  //do something with response

...and the background/event script that listens for broadcasted messages and responds back.

chrome.extension.onMessage.addListener(function(request, sender, sendResponse) {
  //do something
  sendResponse(/* response body */);

I used the word "broadcast" because as you see from the above, there's nothing that indicates who sent the message or what it might contain. You avoid clashes with other installed modules that also use messages by constructing the body of your messages with care. In my case, I construct messages in my content scripts to contain the equivalent of a title in addition to a body so I know I'm the one who sent the message:

{ "message" : "ns_clipBoard", "content" : "my stuff" }

Friday, December 7, 2012

If EOL Started All Over Today, What Would be the Best Approach?

Today I participated in a very engaging conversation with a group of systematists and ecologists who are intensely interested in cataloguing the diversity of life in their neck of the woods. They immediately recognized that such a compilation should contain authoritative content, it should contain links to relevant resources so as not to repeat efforts elsewhere, and it most definitely should be online. In my (perhaps naive) interpretation, it sounded much like the Encyclopedia of Life (EOL), albeit at a smaller, more focused scale.

But has EOL taken a winning approach? Has it sustained the interest it once had? Is it duplicating effort? Is it financially sustainable? Are remarkable, value-added products being built off its infrastructure that would not otherwise be possible? These aren't rhetorical questions. I just don't know. Shouldn't I know by now? Part of the answer will certainly depend on which metric you wish to use. And, these metrics will invariably draw upon the engagement of one audience or another.

Here's an interesting thought experiment:

If EOL had taken a radically different approach at the outset by becoming a taxonomically intelligent index (e.g. a Google-like product, but specifically tuned using a graph such as may be the eventual underpinnings of the Open Tree of Life) instead of serving species pages aggregated from elsewhere, where would it be today? What could have been built from such a "product"?

Thursday, November 8, 2012

Conference Tweets in the Age of Information Overconsumption

Having been a remote Twitter participant in what from all accounts was a successful conference hosted by the Entomological Society of Canada and the Entomological Society of Alberta, I have the luxury of now stepping back with a nice glass of red wine and thinking more deeply about the experience and its implications on the health of science. Dezene Huber has also taken a breath after he participated in person and provided valuable Tweet streams of his own.

The Saturday prior to the conference, I had a "wouldn't be cool if" moment and put my fingers to action on a toy that could listen in on the Tweet streams being generated by conference goers as they prepared for the event, as they were in transit, as they sat in the audience, as they chatted over coffee, and as they celebrated their winnings during the banquet.

My roughshod little experiment was to encourage participants to include scientific names in their streams. After all, names are a very important part of how biology is communicated. I grabbed their Tweets in real-time, fed them into three web services, and stored the results in a relational database. Two of these web services were developed by me and Dmitry Mozzherin at the Marine Biological Laboratory under the NSF-funded Global Names project led by David Patterson. These gave me the tools necessary to answer the questions, "Is this a name?" and "Where is this name in a classification?" The other web service I used was one recently assembled by some brilliant developers at CrossRef that figured out a way to execute rapid searches against their massive database of citations in the primary literature, assembled off the backs of researchers and publishers.

So, while "Ento-Tweeps" tapped a name, I immediately caught it, placed it in a hierarchy, and threw it to CrossRef. Within a split second after a Tweet appeared, I had links to the primary literature and I had some context. These were often amazingly accurate. Here's one that the prolific Morgan Jackson tweeted during Nikolai Tatarnic's paper entitled, "Sexual mimicry and paragenital divergence between sympatric species of traumatically inseminating plant bug":

Now that's useful!


There were occasions where this wasn't so useful. These were examples of what some have called, "Information Overload". But that's a misnomer. We're beginning to understand what this really is. A better term for this (if one were to become dependent and fixated on streams like this) is "Information Overconsumption".

 So, how do we responsibly integrate the power of social media in scientific conferences?

First, draft a light-hearted code of ethics - the same as we've become accustomed to with mobile phones at such events. Turn off the beeps and squawks! Turn off the unnecessary keypress chirps.

Second, as tempting as it may be, DO NOT COMMERCIALIZE THIS! The corporate sector has already found its way into the conference arena, the last pure outlet for the exchange of science. A social media outlet could be a new channel for communication that will be instantly switched off if it were behind a paywall.

Last, treat the messages not as news, but as products. Though the messages are instant, much like a stream of news, they are written by you, the one who has spent years honing your skills and learning your science.

My only hope is that "toys" like mine and the web services upon which they depend improve with time. They MUST help sell your products in a way that does not lead to Information Overconsumption and they MUST add value to the messages you wish to convey. How? That's up to you.

Tuesday, January 3, 2012

Science is a Product in the Wrong Marketplace

Instead of mindlessly watching a movie tonight, I browsed through Google Tech Talks and stumbled upon a spectacularly argued, wonderfully cadenced, and orchestrated Sept 2011 presentation by Kristen Marhaver entitled, "Organizing the world's information by date and author is making Mother Earth Sick".

Her thesis is that science is a product, not a news stream. And, because science is communicated in a self-serving, pay-wall-laden marketplace, its products to outsiders (those who stand to benefit from this knowledge) are paradoxically valueless. Kristen argues that the first steps toward cracking into this marketplace could be to expose the inherently social dimension of science by using modern day social gadgetry. Google+, Twitter and star ratings could reside around the periphery of online PDF reprint viewers. Unfortunately Kristen, this is still the wrong marketplace.

The one place where the social dimension of science is abundantly obvious is the largely unchallenged scientific conference. There are ways for this energetic, youthful, exploratory dialogue to spill out onto the distant screens of those who could benefit. YouTube, Twitter, Google+ could all be used with religion at conferences because for the most part, papers delivered are free from the publisher's grasp. Google Tech Talks and TED talks are spectacularly popular for very good reason. The medium is accessible. Plus, there is ample opportunity to make conferences more accessible and engaging to registrants themselves. How many times have you heard someone deliver a paper who feels the need to introduce his/her co-authors who could not be present or to shamelessly advertise the upcoming paper/poster presentations of their graduate students? The moment someone walks up to the podium, I want all that pushed onto my iPad along with links to their reprints. I'd rather they just get on with it. If their presentation were recorded and later put on YouTube, I'd want the same experience. Sure, links to their reprints would likely throw me up against a brick pay-wall, but I'd already know and appreciate the context.

To take this even further, why not really expose the scientific conference by advertising the downtime? On how many occasions have you gone to a conference, only to share a beer or two in the evening(s) WITH THE COLLEAGUES YOU ALREADY WORK WITH!? Instead, I want a post-conference un-drink. That is, I'd like to advertise my desire to have a drink by posting what I'd like to talk about and then blast the venue into the Twittersphere for members of the public to join me if they felt so inclined. If it's a bust, I'll swallow my pride and go join another one...and I'll bring copies of my reprints.

Monday, November 14, 2011

Realtime Web

I started work on a whimsical presentation I will soon give to the Biodiversity Informatics Group at the Marine Biological Laboratory about the Realtime Web and came-up with the following kooky slide. Felt the urge to share.

Sunday, November 13, 2011

Amazing Web Site Optimizations: Priceless

Quite literally, priceless. As in costs nothing.

I was obsessed with web site optimization these past few weeks, trying to trim off every bit of fat from page render times. As we all know, if a page takes longer than approx. 3-4 seconds to render, then you can expect to lose your audience. Even though expectations for speed vary depending on the end-user's geographic location, having a website that can be equally fast for a user in Beijing is just as important as the experience for a user in California. As might be expected, server hardware typically isn't the bottleneck. Another way of looking at this is to recognize that remarkable boosts in performance can be had on crap hardware. So, this post presents the tools I used to measure web site performance and describes the simple techniques I employed to trim the excess fat.

My drug of choice to measure the effect of every little (or major) tweak has been WebPagetest, a truly invaluable service because I can quickly see where in the world and why my web page suffered. Knowing that it took 'x' ms to download and render a JavaScript file or 'y' ms to do the same for a css file meant I could see with precision what a bit of js or css cleansing does to a user's perception of my web site. I also used Firebug and Yahoo's YSlow, both as FireFox plug-ins. Google Chrome also has a Page Speed extension that I used to produce a few optimized versions of graphics files.

Some tricks I employed to great effect, in order from most to least important:

  1. Make css sprites. The easiest tool I found was the CSS Sprite Generator. Upload a zipped folder of icons and it spits out a download and a css file. Could it be any easier? Making a css sprite eliminates a ton of unnecessary HTTP requests and is by far the most important technique to slash load times.
  2. Minify JavaScript and css. For the longest time, I was using the facile JavaScript Compressor, but the cut/paste workflow became too much of a pain. So, I elected to use some server-side code to do the same: jsmin-php and CssMin. When my page is first rendered, the composite js and css files are made in memory then saved to disk. Upon re-rendering (by anyone), the minified versions are served. Here's the PHP class I wrote that does this for me. Whenever I deploy new code, the cached files are deleted then recreated with a new MD5 hash as file titles.
  3. Properly configured web server. This is especially important for a user's second, third+ visit. You'd be crazy not to take advantage of the fact that a client's browser can cache! I use Apache and here's what I have:

    <Directory "/var/www/SimpleMappr">
    Options -Indexes +FollowSymlinks +ExecCGI
    AllowOverride None
    Order allow,deny
    Allow from all
    DirectoryIndex index.php
    FileETag MTime Size
    <IfModule mod_expires.c>
    <FilesMatch "\.(jpe?g|png|gif|js|css|ico|php|htm|html)$">
    ExpiresActive On
    ExpiresDefault "access plus 1 week"

    Notice that I use the mod_expires module. I also set the FileETag to MTime Size, though this was marginally effective.
  4. Include ALL JavaScript files just before the closing body tag. This boosts the potential for parallelism and the page can begin rendering before all the JavaScript has finished downloading.
  5. Serve JavaScript libraries from a Content Delivery Network (CDN). I use jQuery and serve it from Google. Be weary that on average, it is best to ONLY have 4 external sites from which content will be drawn. This includes static content servers that might be a subdomain associated with your web site. Beyond 3 external domains or subdomains, DNS look-up times outweigh the benefit of parallelism, especially for aged versions of Internet Explorer. Modern browsers are capable of more simultaneous connections, but we cannot (yet) ignore IE. I once served jQueryUI via the Google CDN, but because this was yet another HTTP request, it was slower than had I served it from my own server. So, I now pull jQuery from the Google CDN and I include jQueryUI with my own JavaScript in a single minified file from from my server.
  6. Use a Content Delivery Network. I use CloudFlare because it's free, was configured in 5 minutes and within a day, there was noticeable global improvement in web page speed as measured via WebPagetest. Because I regularly push new code, I use the CloudFlare API to flush their caches whenever I deploy. However, this is largely unnecessary because they do not cache HTML and as mentioned earlier, I make an MD5 hash as my js and css file titles.
So there you have it, I was able to trim 4-6 seconds from a very JavaScript-heavy web site. And, web page re-render speed is typically sub-second from most parts of the world. Because much of the content is proxied through CloudFlare, my little server barely breaks a sweat.

Did I mention that none of the above cost me anything?

Sunday, June 26, 2011

SimpleMappr Embedded

I never had high hopes for SimpleMappr. There are plenty of desktop applications to produce publication-quality point maps. But it turns out, users find these hard to use or are too rich for their pocket books. As a result, my little toy and its API are getting a fair amount of use. I find this greatly encouraging so I occasionally clean-up the code and add a few logical, unobtrusive options.

A number of users appear to want outputs for copy-paste on web pages and not copy-paste into manuscripts, so I just wrote an extension to permit embedding.

Here's one such example using the URL

Happy mapping...