Grimoire Lab is an open source toolkit built on Python, Elasticsearch and Kibana. It taps into GitHub's raw data through Perceval, a module designed for retrieving data from repositories related to software development.

Perceval forwards the data to another tool for filtering, the so called Sorting Hat, responsible for managing and merging identities that correspond to the same real person/commiter and potentially come from different sources, before finally rendering the data manageable and accessible through rich UI Kibiter dashboards. Kibiter, a fork of Kibana, is what enables the user to create and edit visualizations as well as perform queries facilitated by the underlying Elasticsearch REST APIs.

Note the small-print; Perceval is "specialized in retrieving data from repositories related to software development". This means it is not restricted just to GitHub but can attach to a variety of disparate data sources able to host either source code, or code reviews, mailing lists and forums, meetups, tickets, issues, chats, you name it. Concrete examples are  Bugzilla, JIRA, Gerrit, mbox, pipermail, StackExchange and Discourse. In fact makers Bitergia, offer enterprise packages for businesses which want to hook up their datasources to extract  information.

When Perceval is applied to GitHub you end up with a powerful Open Source Community Analytics platform which can answer a multitude of inquiries revolving around an open source community; "How big? How many?",  "Where from? Organizations? Gender?", "How old? New one? Inactive ones?", "How fast?". In other words, through metrics like these and many others, Grimoire can provide indications and measurements on a project's success. 

As proof of concept, in Grimoire's repository we find a good use case on how to use Perceval to retrieve data from a GitHub project (git repository and associated GitHub issues), upload it to Elasticsearch, and produce a Kibana-based dashboard with it. You can find detailed instructions on how to set it up here, but suffice to say that the procedures are mostly Python related :

You first create and activate virtualenv for Python:

Then install the necessary dependencies:

(gitlab) $ pip install beautifulsoup4
    (gitlab) $ pip install python-dateutil
    (gitlab) $ pip install requests
    (gitlab) $ pip install six

and so on.

To make it even easier, Bitergia has setup Cauldron exposing Grimoire as a service so that you can monitor your GitHub repo without having to set up anything! Each GitHub user may ask for up to five dashboards, each corresponding to a GitHub owner (user or organization).

For example let's have a look at the Eclipse Foundation projects' dashboard, which monitors the foundation's mailing lists as well as its Git, Gerrit and Bugzilla repositories.

In the top Git panel we can immediately get a glimpse at some initial statistics, like that there were 136,776 commits since 2015 and there are 2,146 authors (contributors), with Thomas Schindl the most prolific one. Subsequently following the Thomas Schindl node we find out that he is contributing to the technology.efxclipse and eclipse.platform.ui projects, and even see the hours of the day that he makes most commits! Needless to say, following node after node, we get into more depth in examining all project aspects.

Best of all and as already said, Grimoire is open source to the core and itself available on GitHub. As it isn't a single product but an ecosystem comprising of distinct components, each one comes under its own license.  Perceval and Sorting Hat are under the GNU General Public License (GPL), version 3, while Kibiter is under Elasticsearch's Apache License, Version 2.0. so that you can use them in your own setups and applications. 

More Information

Grimoire

Grimoire on GitHub

Cauldron

Related Articles

GitHub Extends Developer Program

GitHub Platform and Community Improvements

GitHub Octoverse Reveals The State Of Open Source

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info