It seems amazing but true that one of the disadvantages of almost any JavaScript generated content is that it stands a very good chance of not being properly indexed by Google, or anyone else for that matter. 

Programmers may like clever clientside techniques, but if the web site or app relies on being crawled correctly to send it traffic then this presents additional problems. Googlebot, Google's webcrawler, has been doing its best to take account of JavaScript's effect on page content, but it seems to be a tough problem. 

Recently the Webmaster Central Blog detailed some of the efforts that Google has been making to try and make sense of JavaScript. 

"In order to solve this problem, we decided to try to understand pages by executing JavaScript. It’s hard to do that at the scale of the current web, but we decided that it’s worth it. We have been gradually improving how we do this for some time. In the past few months, our indexing system has been rendering a substantial number of web pages more like an average user’s browser with JavaScript turned on."

This is good news and perhaps it makes the idea of creating single page apps more appealing. However the blog goes on to outline possible problems. The majority are obvious - like don't block Googlebot's access to files needed to render the page. The only one that should worry you is:

"Sometimes the JavaScript may be too complex or arcane for us to execute, in which case we can’t render the page fully and accurately."

Has anyone any idea what Google is getting at? If the JavaScript renders in a modern browser surely that should be enough, no matter how arcane or complex it is?

The usual solution to making sure that JavaScript based apps are crawled properly is to create an HTML version that can be fed to GoogleBot but clearly this is a waste of time and in many cases it results in the minimum content being crawled. 

It is good that Googlebot is trying to interpret JavaScript, but what about other web crawlers and social media? The need to generate HTML versions for reduced level interaction with robots is a big problem for advanced website design and it needs to be solved. It is about time that all robot interaction with web pages did the job at the level of a full browser and it is no longer enough to simply follow the links.

Finally, the blog post promises that there will be help on Webmaster Tools to understand how Googlebot sees complex web pages - it can't come soon enough. 

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info