Yahoo Open Sources Vespa
Yahoo Open Sources Vespa
Written by Kay Ewbank   
Friday, 20 October 2017

The big data serving system that powers Yahoo and Flickr has been open sourced by Oath, the parent company to Yahoo.  

Most developers will be familiar with Hadoop, which Yahoo made open source in 2006. Vespa is another element in Yahoo's data management suite, the data processing and serving engine, and it too is being made available as open source on GitHub.

 

vespa

 

Vespa deals with the problem of serving results to end users where there's a need to make computations over data while a user is waiting, as is the case with applications that feature search, recommendation, and personalization. Vespa means you can create applications that compute responses to user requests, over large datasets, at real time and at internet scale.

Oath (the group of Verizon that owns Yahoo) uses Vespa across many different Oath brands – including Yahoo.com, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Gemini and Flickr. The system handles billions of daily requests over billions of documents while responding to search queries, making recommendations, and providing personalized content and advertisements. It also serves over 3 billion native ad requests per day via Yahoo Gemini.

Vespa can be used to select content items using SQL-like queries and text search, and to organize the matches to generate data-driven pages. It can rank the matches using either hard-coded or machine-learned relevance models. The results are served with response times in the lows milliseconds, and it can write data in real-time at a rate of thousands of times per second per node.

Vespa distributes data and computation over many machines without any single master as a bottleneck. You can feed data and models of any size to the serving system, then make the final computations at request time. It can be run on premises or in the cloud, and there are both Docker images and rpm packages for Vespa.  You can read how to build and manage Vespa systems, and the Vespa blog is running a series of posts on how to make the most of Vespa.

 

vespa

More Information

Vespa Blog 

Vespa On GitHub 

Related Articles

Yahoo In Retrenchment Again

Yahoo Shuts Services

Yahoo Opens Up Flickr Deep Learning

Yahoo Releases Record Machine  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

 

Banner


AWS DeepLens Available for Pre-Order
25/05/2018

A video camera for developers that is enabled for deep learning is available for pre-order on Amazon.



Udacity's Android Developer Nanodegree Updated To Android Architecture Components
28/05/2018

Udacity's Android Developer Nanodegree gets up to speed with Android Architecture Components with an important update. This recognizes that the components are growing in popularity and can  even  [ ... ]


More News

 

justjsquare

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Friday, 20 October 2017 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2018 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.