Yahoo Open Sources Vespa
Yahoo Open Sources Vespa
Written by Kay Ewbank   
Friday, 20 October 2017

The big data serving system that powers Yahoo and Flickr has been open sourced by Oath, the parent company to Yahoo.  

Most developers will be familiar with Hadoop, which Yahoo made open source in 2006. Vespa is another element in Yahoo's data management suite, the data processing and serving engine, and it too is being made available as open source on GitHub.

 

vespa

 

Vespa deals with the problem of serving results to end users where there's a need to make computations over data while a user is waiting, as is the case with applications that feature search, recommendation, and personalization. Vespa means you can create applications that compute responses to user requests, over large datasets, at real time and at internet scale.

Oath (the group of Verizon that owns Yahoo) uses Vespa across many different Oath brands – including Yahoo.com, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Gemini and Flickr. The system handles billions of daily requests over billions of documents while responding to search queries, making recommendations, and providing personalized content and advertisements. It also serves over 3 billion native ad requests per day via Yahoo Gemini.

Vespa can be used to select content items using SQL-like queries and text search, and to organize the matches to generate data-driven pages. It can rank the matches using either hard-coded or machine-learned relevance models. The results are served with response times in the lows milliseconds, and it can write data in real-time at a rate of thousands of times per second per node.

Vespa distributes data and computation over many machines without any single master as a bottleneck. You can feed data and models of any size to the serving system, then make the final computations at request time. It can be run on premises or in the cloud, and there are both Docker images and rpm packages for Vespa.  You can read how to build and manage Vespa systems, and the Vespa blog is running a series of posts on how to make the most of Vespa.

 

vespa

More Information

Vespa Blog 

Vespa On GitHub 

Related Articles

Yahoo In Retrenchment Again

Yahoo Shuts Services

Yahoo Opens Up Flickr Deep Learning

Yahoo Releases Record Machine  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

 

Banner


Bing Maps Improves BirdsEye Presentation
23/10/2017

There's an updated version of the Bing Maps Web Control with a number of new features. The developers describe the additions as 'smaller but highly requested'. 



Visual Search Adopted by eBay
08/11/2017

Artificial Intelligence is rapidly taking over all aspects of our lives, from calorie counting, to personal assistance and financial advice. Shopping is no exception. Now, with eBay's new powered-by-A [ ... ]


More News

 

 
 

 

blog comments powered by Disqus

Last Updated ( Friday, 20 October 2017 )
 
 

   
Banner
Banner
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.