The big data serving system that powers Yahoo and Flickr has been open sourced by Oath, the parent company to Yahoo.  

Most developers will be familiar with Hadoop, which Yahoo made open source in 2006. Vespa is another element in Yahoo's data management suite, the data processing and serving engine, and it too is being made available as open source on GitHub.

Vespa deals with the problem of serving results to end users where there's a need to make computations over data while a user is waiting, as is the case with applications that feature search, recommendation, and personalization. Vespa means you can create applications that compute responses to user requests, over large datasets, at real time and at internet scale.

Oath (the group of Verizon that owns Yahoo) uses Vespa across many different Oath brands – including Yahoo.com, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Gemini and Flickr. The system handles billions of daily requests over billions of documents while responding to search queries, making recommendations, and providing personalized content and advertisements. It also serves over 3 billion native ad requests per day via Yahoo Gemini.

Vespa can be used to select content items using SQL-like queries and text search, and to organize the matches to generate data-driven pages. It can rank the matches using either hard-coded or machine-learned relevance models. The results are served with response times in the lows milliseconds, and it can write data in real-time at a rate of thousands of times per second per node.

Vespa distributes data and computation over many machines without any single master as a bottleneck. You can feed data and models of any size to the serving system, then make the final computations at request time. It can be run on premises or in the cloud, and there are both Docker images and rpm packages for Vespa.  You can read how to build and manage Vespa systems, and the Vespa blog is running a series of posts on how to make the most of Vespa.

More Information

Vespa Blog 

Vespa On GitHub 

Related Articles

Yahoo In Retrenchment Again

Yahoo Shuts Services

Yahoo Opens Up Flickr Deep Learning

Yahoo Releases Record Machine  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info