Apache DataFu Pig
Apache DataFu Pig is a collection of user-defined functions and macros for working with large scale data in Apache Pig.
It has a number of useful functions available. This guide provides examples of how to use these functions and serves as an overview for working with the library.
- Statistics: median, quantiles, variance
- Bag Operations: join, prepend, append, count items, concat
- Set Operations: set intersection, union, difference
- Sessions: sessionize streams of data
- Sampling: simple random sample with/without replacement, weighted sample, sample by keys
- Hashing: SHA and MD5
- Link Analysis: PageRank
- Assorted Macros: deduplication of tables, human-readable diffs and more
- More Tips and Tricks
There are also Javadocs available for all UDFs in the library. We continue to add
UDFs to the library. If you are interested in helping out please follow the Contributing
The current version of DataFu has been tested against Pig 0.14.0. DataFu should be compatible with some older versions of Pig, however we do not do any sort of testing with prior versions of Pig and do not guarantee compatibility.
Our policy is to test against the most recent version of Pig whenever we release and make sure DataFu works with that version.