Facebook has developed a security-focused framework called ThreatData that it says will make it simpler to manage a range of online threats.

The framework attempts to provide a single source of information about threats on the Internet. According to Facebook's Internet Threat Researcher Mark Hammell, the idea behind it is:

“Given the pace of criminals today, one of the hard parts is actually keeping track of all the data related to malware, phishing, and other risks. We wanted an easier way to organize our work and incorporate new threat information we receive so that we can do more to protect people.”

The framework Facebook developers have created lets the company import information about “badness” on the Internet in arbitrary formats, store it efficiently, and making the data accessible for both real-time defensive systems and long-term analysis.

The ThreatData framework has three high-level parts: feeds, data storage, and real-time response.

Feeds collect data from a specific source and are implemented via a light-weight interface. The data can be in imported in most formats, and the feed transforms it into a simple schema that is capable of storing not only the basics of the threat (e.g., evil-malware-domain.biz) but also the context in which it was bad. The added context is used in other parts of the framework to make more informed, automatic decisions.

Once transformed, the data is fed into both Hive and Scuba; Hive is then used to answer questions based on long-term data such as “Have we ever seen this threat before?” and “What type of threat is more prevalent from our perspective: malware or phishing?”, while Scuba is used for more immediate analysis along the lines of “What new malware are we seeing today?” and “Where are most of the new phishing sites?”.

Facebook has also developed a processor to examine the data at the time of logging and act on each of these new threats. Hammell gave examples implemented so far including the fact that all malicious URLs collected from any feed are sent to the same blacklist used to protect people on facebook.com; and that interesting malware file hashes are automatically downloaded from known malware repositories, stored, and sent for automated analysis.

The analysis has highlighted some trends in malware, including a spam campaign aimed at feature phones that was capable of stealing a victim's address book, sending premium SMS spam, and using the phone's camera to take pictures. The framework also lets Facebook view where threats are coming from, arranged by type of attack, time, and frequency. The notes include a worldwide heat map showing malicious and victimized IP addresses, with a pie chart showing similar results for the U.S. by ISP.

In his post Hamell comments:

“Discoveries and detection capabilities like these are just the tip of the iceberg . We’re constantly finding new ways to improve and extend the ThreatData framework to encompass new threats and make smarter decisions with the ones we’ve already identified.”