Speedometer - Meet Google's New JavaScript Benchmark
Written by Mike James   
Thursday, 13 April 2017

Benchmarking is both essential and a distortion. The trouble is that once there is a benchmark that anyone pays any attention to, the tendency is to tweak for a high score - even if this results in a poorer real world performance. 

 

speedometer

 

Google's V8 JavaScript engine is often the one to beat, or rather to try to beat. The standard way of proving that your JavaScript engine was fast was to run it against the Google-produced Octane benchmark.

As already mentioned, the problem was that as soon as Octane was taken seriously it started to be misused to prove whatever was needed. The situation is a bit like the diesel-gate scandal that saw motor manufacturers modifying the behavior of their engines to reduce emissions when they detected that a test was in progress. In this case. though. the engines were JavaScript engines and in most cases their behaviour was permanently modified to produce the result at the expense of real world performance.

Octane contained cut down real world code and you might have assumed that this would make it more relevant, but no.  As the Chromium blog explains:

At the beginning of last year, the V8 team started to measure performance with higher fidelity by instrumenting snapshots of popular web pages such as Reddit, Twitter, Facebook, and Wikipedia. This analysis revealed that while peak performance benefits certain types of large web applications, browsing typical websites relies more on “startup” performance, or the speed it takes to start running script. Using insights gleaned from this real-world performance data, the V8 team implemented optimizations which improved mean page load between Chrome 49 and Chrome 56 by 10-20%, depending on CPU architecture.

From the data gathered the team put together a new benchmark - Speedometer - with added automotive reference. It includes chunks of code from common web frameworks such as React, Angular, Ember and jQuery. The important point is that while Octane didn't respond to the startup optimizations, Speedometer did - suggesting that optimizing with respect to Speedometer might yield real world speedups. As a consequence:

Given the plateau of Octane scores across web browsers and the over-optimization of peak performance, we decided to retire the benchmark as a general-purpose measure of real-world JavaScript performance.

octane

What do we conclude?

The first is that benchmarking isn't easy and never has been easy. An established benchmark that becomes detatched from the real world workload distorts the optimization made. The Google V8 team seem to have the right idea:

Going forward, we plan to ship more JavaScript performance improvements for new patterns of script appearing on the web, including modern libraries, frameworks, and ES2015+ language features. By measuring real-world websites rather than traditional benchmarks, we can better optimize JavaScript patterns that matter most to users and developers.

It is amazing how much optimization can be achieved even for a complex dynamic language like JavaScript. Perhaps, if we work hard at it, innovations like asm.js and web assembly just won't be needed!

You can try Speedometer out on your hardware with the browser of your choice at: Speedometer benchmark.

 

speedo1

 

More Information

Speedometer benchmark.

Real-world JavaScript performance

Related Articles

TurboFan For Enhanced JavaScript Performance 

Chrome V8 JavaScript - Ever Faster 

Asm.js Gets Faster 

Octane 2.0 Released 

Google's Octane - New JavaScript Benchmark

JavaScript Is Slow

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

 

 

Banner


Crazy Clocks
10/03/2024

It's that time again when the clocks change and  time is of the essence and I indulge my interest in crazy clocks. I am always surprised that there are still new ideas for how to display the time [ ... ]



Quantum Company Wins Digital Startup Contest
03/03/2024

Quantum specialists Qilimanjaro is the winner of this year's Four Years From Now competition. The award was made at this year's Mobile World Congress (MWC) in Barcelona.


More News

 

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

 

 

 

 

Last Updated ( Thursday, 13 April 2017 )