Hortonworks has announced an initiative with IBM and Red Hat in the drive to make Hadoop cloud-native and able to run well in hybrid environments.

While the current versions of Hadoop can be used in cloud environments, Hadoop itself largely ignores the fact it's being used in the cloud. Despite this, it is increasingly deployed in the cloud, hence the new announcement. Hortonworks has also released a roadmap of when Hadoop will be ready to run in hybrid environments.

Hortonworks says that it has been working towards the goal of hybrid running, which according to them requires:

• Cloud-native Hadoop for public cloud – delivered with Hortonworks Data Platform (HDP) and Hortonworks DataFlow (HDF) on IaaS.
• Data flow and management to and from the edge – delivered with HDF, and specifically with MiNiFi.
• Consistent security and data governance across all tiers – delivered with DPS.
• A consistent architecture in the cloud and on-premises. This is the last mile.

DPS (Data Plane Services) is a catalog of services that run as plug-ins for tasks such as lifecycle management, replication management, access control, and data flow management. 

The initiative with IBM and Red Hat is aimed at creating the consistent architecture no matter where Hadoop is running. This requires storage to be decoupled from the computing environment, and the use of containerized computing resources to ensure software isolation. Services should be shared across all tiers to help with governance and security, and tools provided for managing services and workloads to allow spin-up/down programmatically. The final element of this is to be able to designate workloads that are specific to particular uses such as as EDW or data science rather than sharing everything in a multi-tenant Hadoop cluster.

As the initial phase of the initiative, Hortonworks, Red Hat and IBM will work together to optimize Hortonworks Data Platform, Hortonworks DataFlow, Hortonworks DataPlane and IBM Cloud Private for Data for use on Red Hat OpenShift, an enterprise container and Kubernetes application platform. This should make it possible to develop and deploy containerized big data workloads. IBM and Hortonworks will continue working to integrate services offered through Hortonworks DataPlane with IBM Cloud Private for Data.

More Information

Hortonworks

Related Articles

Hadoop 3 Adds HDFS Erasure Coding

Hadoop 2.9 Adds Resource Estimator

Hadoop Adds In-Memory Caching

Hadoop SQL Query Engine Launched

Hadoop 2 Introduces YARN

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info