Apache Hadoop goes Enterprise

The roughly 1,000 participants at the 3rd Hadoop Summit in Santa Clara, California show how popular and important the Apache Hadoop project is becoming among developers. The two previous conferences drew 300 (2008) and 600 (2009) visitors. The next major step for the framework – created in 2005 by Yahoo and handed off to the Apache Software Foundation (ASF) in 2006 – seems to be greater market readiness. At least, that's what a number of recent announcements would seem to indicate.

Hadoop facilitates the development of distributed applications that conduct complex calculations and processing on large amounts of data. The open source software implements Google's MapReduce algorithm and is currently used by Amazon, AOL, Facebook and Yahoo's search engine.

Yahoo!, which says it currently has 38,000 Hadoop Web servers running, has announced two products. The first is "Hadoop with Security", an integration with the Kerberos authentication service to provide safe access to and ensure the safe processing of mission-critical data. The second is Oozie, an open source workflow for the management of jobs on Hadoop installations. The Hadoop Distributed File System (HDFS), the high-level Pig language and the MapReduce implementation are included; Oozie is integrated with the "Hadoop with Security" package.

Hadoop service provider Cloudera has also released an enterprise version of the open source Cloudera Distribution for Hadoop (CDH) presented in March. For the premium Cloudera Enterprise, subscribers also receive additional tools for the monitoring, management and administration of Hadoop installations and services. Version 3 of CDH, which was also just announced, offers a de-facto standard for data management platforms based on Hadoop.

See also:

• Google grants license for Apache Hadoop, a report from The H.

(crve)