Getting Started
DataFu Spark Docs
DataFu Pig Docs
DataFu Hourglass Docs
Community
Apache Software Foundation
Apache DataFu Pig
Guide
Apache DataFu Pig is a collection of user-defined functions and macros for working with large scale data in Apache Pig. It has a number of useful functions available. This guide provides examples of how to use these functions and serves as an overview for working with the library.
- Statistics: median, quantiles, variance
- Bag Operations: join, prepend, append, count items, concat
- Set Operations: set intersection, union, difference
- Sessions: sessionize streams of data
- Sampling: simple random sample with/without replacement, weighted sample, sample by keys
- Hashing: SHA and MD5
- Link Analysis: PageRank
- Assorted Macros: deduplication of tables, human-readable diffs and more
- More Tips and Tricks
There are also Javadocs available for all UDFs in the library. We continue to add UDFs to the library. If you are interested in helping out please follow the Contributing guide.
Pig Compatibility
The current version of DataFu has been tested against Pig 0.14.0. DataFu should be compatible with some older versions of Pig, however we do not do any sort of testing with prior versions of Pig and do not guarantee compatibility. Our policy is to test against the most recent version of Pig whenever we release and make sure DataFu works with that version.
Blog Posts
- Introducing DataFu
- DataFu: The WD-40 of Big Data
- DataFu 1.0
- A Look at PayPal's Contributions to DataFu
Slides
Videos
Apache DataFu, DataFu, Apache Pig, Apache Hadoop, Hadoop, Apache, and the Apache feather logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.