Yahoo! creates independent Hadoop company

Yahoo! has announced that in conjunction with Benchmark Capital it has created an independent spin off company, HortonWorks, which will focus on Apache Hadoop technology. Yahoo! hopes that the new company will help accelerate adoption of Hadoop by making it more robust and easier to deploy in enterprises.

The new company will be lead by Eric Baldeschwieler, formerly VP of software engineering for the Hadoop team at Yahoo!, who believes that "within five years, more than half the world's data will be stored in Apache Hadoop", and that the technology will "quickly become the de facto platform for managing big data." Hortonworks will begin life with many of the key architects and developers from Yahoo!'s own Hadoop engineering team and has brought on Rob Bearden, formerly COO at SpringSource and JBoss, to handle the company's commercial affairs.

Raymie Stata, Yahoo!'s CTO, wrote in a blog posting that the Hortonworks focus will be on the commercialisation and standardisation of the technology. Although the spinoff is being created, Yahoo! plans to maintain its own investment in Hadoop as it runs a 40,000 server Hadoop installation processing five billion jobs a month supporting Yahoo!'s ad network and anti-spam for Yahoo! Mail.

Stata reassures the Apache community that Yahoo! "will continue passing along our knowledge and code to you through Apache", and noted that it aims to ensure that Hortonworks will have the same philosophy of sharing. Hortonwork's manifesto cites open source as the third of its three principles; the other principles are "Architecting the Future of Big Data" and "Solving Real-World Business Challenges".

(djwm)