Benchmarking is both essential and a distortion. The trouble is that once there is a benchmark that anyone pays any attention to, the tendency is to tweak for a high score - even if this results in a poorer real world performance. 

Google's V8 JavaScript engine is often the one to beat, or rather to try to beat. The standard way of proving that your JavaScript engine was fast was to run it against the Google-produced Octane benchmark.

As already mentioned, the problem was that as soon as Octane was taken seriously it started to be misused to prove whatever was needed. The situation is a bit like the diesel-gate scandal that saw motor manufacturers modifying the behavior of their engines to reduce emissions when they detected that a test was in progress. In this case. though. the engines were JavaScript engines and in most cases their behaviour was permanently modified to produce the result at the expense of real world performance.

Octane contained cut down real world code and you might have assumed that this would make it more relevant, but no.  As the Chromium blog explains:

At the beginning of last year, the V8 team started to measure performance with higher fidelity by instrumenting snapshots of popular web pages such as Reddit, Twitter, Facebook, and Wikipedia. This analysis revealed that while peak performance benefits certain types of large web applications, browsing typical websites relies more on “startup” performance, or the speed it takes to start running script. Using insights gleaned from this real-world performance data, the V8 team implemented optimizations which improved mean page load between Chrome 49 and Chrome 56 by 10-20%, depending on CPU architecture.

From the data gathered the team put together a new benchmark - Speedometer - with added automotive reference. It includes chunks of code from common web frameworks such as React, Angular, Ember and jQuery. The important point is that while Octane didn't respond to the startup optimizations, Speedometer did - suggesting that optimizing with respect to Speedometer might yield real world speedups. As a consequence:

Given the plateau of Octane scores across web browsers and the over-optimization of peak performance, we decided to retire the benchmark as a general-purpose measure of real-world JavaScript performance.

What do we conclude?

The first is that benchmarking isn't easy and never has been easy. An established benchmark that becomes detatched from the real world workload distorts the optimization made. The Google V8 team seem to have the right idea:

Going forward, we plan to ship more JavaScript performance improvements for new patterns of script appearing on the web, including modern libraries, frameworks, and ES2015+ language features. By measuring real-world websites rather than traditional benchmarks, we can better optimize JavaScript patterns that matter most to users and developers.

It is amazing how much optimization can be achieved even for a complex dynamic language like JavaScript. Perhaps, if we work hard at it, innovations like asm.js and web assembly just won't be needed!

You can try Speedometer out on your hardware with the browser of your choice at: Speedometer benchmark.

More Information

Speedometer benchmark.

Real-world JavaScript performance

Related Articles

TurboFan For Enhanced JavaScript Performance 

Chrome V8 JavaScript - Ever Faster 

Asm.js Gets Faster 

Octane 2.0 Released 

Google's Octane - New JavaScript Benchmark

JavaScript Is Slow

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info