Apache DataFu™

Apache DataFu-Spark 2.0.0 Released

Eyal Allweil

I'd like to announce the release of Apache DataFu-Spark 2.0.0.

This version is the first to support Spark 3.x. In this release, Spark versions 3.0.0 to 3.1.3 are supported.

The four classes in SparkUDAFs - MultiSet, MultiArraySet, MapMerge and CountDistinctUpTo are deprecated. Instead of them, there are new versions which use the Spark Aggregator API. The deprecated versions will be removed in DataFu 2.1.0.


  • Spark 3.0.0 - 3.1.3 supported (DATAFU-169)
  • New Aggregators replace deprecated UserDefinedAggregateFunction (DATAFU-173)

Breaking changes

  • Spark 2.x no longer supported

The source release can be obtained from:


Artifacts for DataFu are published in Apache's Maven Repository:


Please visit the Download page for instructions on building from source or retrieving the artifacts in your build system.