Apache DataFu™

Apache DataFu

Apache DataFu™ is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.

It consists of two libraries:

  • Apache DataFu Pig: a collection of user-defined functions for Apache Pig
  • Apache DataFu Hourglass: an incremental processing framework for Apache Hadoop in MapReduce

Apache DataFu Pig

Apache DataFu Pig is a collection of useful user-defined functions for data analysis in Apache Pig. This library was open sourced in 2010 and continues to receive contributions, having reached 1.0 in September, 2013. It has been used by production workflows at LinkedIn since 2010. It is also included in Cloudera's CDH and Apache Bigtop. All of the UDFs are unit tested to ensure quality.

Check out the Getting Started guide to learn more.

Apache DataFu Hourglass

Apache DataFu Hourglass is a library for incrementally processing data using Hadoop MapReduce. This library was inspired by the prevelance of sliding window computations over daily tracking data at LinkedIn. Computations such as these typically happen at regular intervals (e.g. daily, weekly), and therefore the sliding nature of the computations means that much of the work is unnecessarily repeated. DataFu's Hourglass was created to make these computations more efficient, yielding sometimes 50-95% reductions in computational resources.

Work on this library began in early 2013, which led to a paper presented at IEEE BigData 2013. It is currently in production use at LinkedIn.

Check out the Getting Started guide to learn more.


Apache DataFu is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.