googlebot

GoogleBot ExpressJs

This module implements a middleware for express that allows to render a full Html/JS/Css version of a
page when JS is not available in the client and the site relies heavily on it to render the site, like
when using ember/angular/jquery/backbone; I needed to code this for work to be able to deliver a
SEO friendly version of the website to the Google Crawler, and found no solution available.

Docs

Google Crawler will
attempt a different url when certain characteristics are met, make sure your site complains with them, you have
two options for this
* You must replace your # with #!
* You can add a meta tag to your layout this must be done server side,
if this is not found in the initial response it won't work
Later we will try to figure out the user agent and make it available to more crawlers, or prevent crawling.

Google will replace the hashbang (or the url) with ?_escaped_fragment_= and append the rest of the url there
and expects a different, completely rendered version of the site, the middleware will realize when the request
has this and instead of retrieving the normal response it will return the full rendered version that phantomJS
creates.

The url fragment that triggers the rendering in phantom can be customized, and something can be appended to it
to create conditionals that will restrict crawling or hide certain parts from Google, this too can be
customized.

I tried to make it as custom as possible to create different uses withouth having to modify the core files,
so you can even serve static files from a different server if it was
the case; since this is technically a proxy you can use it for many things. Pull request are welcome and
encouraged tho.

Getting Started

$3

Since we are probably hosting this in a virtual machine installing a new program might not be as trivial as
installing it in our shiny macbooks. This is how you download phantom, uncompress it and add the binaries to
the path.

cd ~/
mkdir phantom
cd phantom
wget https://phantomjs.googlecode.com/files/phantomjs-1.9.2-linux-x86_64.tar.bz2
sudo mv phantomjs-1.9.2-linux-x86_64.tar.bz2 /usr/local/share/.
cd /usr/local/share
sudo tar -xf phantomjs-1.9.2-linux-x86_64.tar.bz2
sudo ln -s /usr/local/share/phantomjs-1.9.2-linux-x86_64 /usr/local/share/phantomjs
sudo ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/local/bin/phantomjs

$3

Remember this is middleware for express, I don't know how it works in other frameworks, if you do fork it and
make it better :)

There's probably no point on installing globally, but if you wish to it will install

npm install --save googlebot

To install locally, or add googlebot in your package.json

$3

In your server.coffee o server.js when you launch the server add the line for googlebot, tada!

app.use googlebot {option:value}

if javascript

app.use(googlebot({option:'value', option2:'othervalue'}));

More complete example

googlebot = require 'googlebot'
express = require 'express'

app = module.exports = express()
app.configure ->
app.set 'views', __dirname + '/views'
app.use googlebot {delay: 5000, canonical: 'http://dvidsilva.com'}
app.use (req, res) ->
res.render 'app/index'

app.startServer = (port) ->
app.listen port, ->
console.log 'Express server started on port %d in %s mode!',
port, app.settings.env

Options

$3

default: true

whether or not to respond to google requests or request that meet a particular requirement(someday)

$3

default: '?_escaped_fragment_='

Which string in the url triggers the phantom rendering instead

$3

default: '&phantom=true'

Add something to the new request, I use to prevent Google from seeing certain stuff

$3

default: 1000

Number of miliseconds to wait for the page to render before sending the request

$3

default: 'http'

In case you want to redirect the request to a different one

$3

default: undefined

In case you want to redirect phantomJS requests to a different host even, where you store the static
files or something

$3

default: undefined

ref
specify the preferred host for google to associate the page resulting, a header will be sent to tell Google
which url you rather show to the people searching for you

$3

default: function(){};

(currently not supported) the idea is to allow you to add more client side javascript that phantomJS will
execute before returning the results to Google withouth having to modify the module. An example could be that
you don't want to have empty alt tags in your images, because is bad SEO so you can do

$('img').each(function(){ $(this).attr('alt',$(this).attr('src')); });

Dependencies and notes

* You need to install PhantomJS and make it available in the PATH
* Node Phantom is used to communicate between
Node and the Phantom Browser
* ExpressJS is a Node web application framework and this GoogleBot is a middleware
for it, if you're using a different framework it might or not work, I have no idea, but at least you can
get some inspiration and copy what's useful
* Google Ajax Crawling Google
will attempt a different url if certain characteristics are met, you must be complaint with them

Thanks to

Crawlme That implements a simmilar module
to use with ZombieJS instead of Phantom

Contact

Use github for issues or questions so everybody can benefit.

$3

@dvidsilva