Speedometer - Meet Google's New JavaScript Benchmark
Speedometer - Meet Google's New JavaScript Benchmark
Written by Mike James   
Thursday, 13 April 2017

Benchmarking is both essential and a distortion. The trouble is that once there is a benchmark that anyone pays any attention to, the tendency is to tweak for a high score - even if this results in a poorer real world performance. 

 

speedometer

 

Google's V8 JavaScript engine is often the one to beat, or rather to try to beat. The standard way of proving that your JavaScript engine was fast was to run it against the Google-produced Octane benchmark.

As already mentioned, the problem was that as soon as Octane was taken seriously it started to be misused to prove whatever was needed. The situation is a bit like the diesel-gate scandal that saw motor manufacturers modifying the behavior of their engines to reduce emissions when they detected that a test was in progress. In this case. though. the engines were JavaScript engines and in most cases their behaviour was permanently modified to produce the result at the expense of real world performance.

Octane contained cut down real world code and you might have assumed that this would make it more relevant, but no.  As the Chromium blog explains:

At the beginning of last year, the V8 team started to measure performance with higher fidelity by instrumenting snapshots of popular web pages such as Reddit, Twitter, Facebook, and Wikipedia. This analysis revealed that while peak performance benefits certain types of large web applications, browsing typical websites relies more on “startup” performance, or the speed it takes to start running script. Using insights gleaned from this real-world performance data, the V8 team implemented optimizations which improved mean page load between Chrome 49 and Chrome 56 by 10-20%, depending on CPU architecture.

From the data gathered the team put together a new benchmark - Speedometer - with added automotive reference. It includes chunks of code from common web frameworks such as React, Angular, Ember and jQuery. The important point is that while Octane didn't respond to the startup optimizations, Speedometer did - suggesting that optimizing with respect to Speedometer might yield real world speedups. As a consequence:

Given the plateau of Octane scores across web browsers and the over-optimization of peak performance, we decided to retire the benchmark as a general-purpose measure of real-world JavaScript performance.

octane

What do we conclude?

The first is that benchmarking isn't easy and never has been easy. An established benchmark that becomes detatched from the real world workload distorts the optimization made. The Google V8 team seem to have the right idea:

Going forward, we plan to ship more JavaScript performance improvements for new patterns of script appearing on the web, including modern libraries, frameworks, and ES2015+ language features. By measuring real-world websites rather than traditional benchmarks, we can better optimize JavaScript patterns that matter most to users and developers.

It is amazing how much optimization can be achieved even for a complex dynamic language like JavaScript. Perhaps, if we work hard at it, innovations like asm.js and web assembly just won't be needed!

You can try Speedometer out on your hardware with the browser of your choice at: Speedometer benchmark.

 

speedo1

 

More Information

Speedometer benchmark.

Real-world JavaScript performance

Related Articles

TurboFan For Enhanced JavaScript Performance 

Chrome V8 JavaScript - Ever Faster 

Asm.js Gets Faster 

Octane 2.0 Released 

Google's Octane - New JavaScript Benchmark

JavaScript Is Slow

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

 

 

Banner


Are Developer Bootcamps Dying Out?
16/08/2017

The announcement of the impending closure of Dev Bootcamp, and news that The Iron Yard had already stopped accepting new registrations  has led to some speculation about the demise of coding boot [ ... ]



Three New Experiments for Firefox Test Pilot
03/08/2017

Firefox Test Pilot is an opt-in program where you can have early access to features designed to be Firefox add-ins and extensions while they are in the early stages of development. This week three new [ ... ]


More News

 

 
 

 

blog comments powered by Disqus

 

 

 

 

Last Updated ( Thursday, 13 April 2017 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.