apache-spark Shared Variables Accumulators
Example
Accumulators are write-only variables which can be created with SparkContext.accumulator:
val accumulator = sc.accumulator(0, name = "My accumulator") // name is optional
modified with +=:
val someRDD = sc.parallelize(Array(1, 2, 3, 4))
someRDD.foreach(element => accumulator += element)
and accessed with value method:
accumulator.value // 'value' is now equal to 10
Using accumulators is complicated by Spark's run-at-least-once guarantee for transformations. If a transformation needs to be recomputed for any reason, the accumulator updates during that transformation will be repeated. This means that accumulator values may be very different than they would be if tasks had run only once.
Note:
- Executors cannot read accumulator's value. Only the driver program can read the accumulator’s value, using its value method.
- It is almost similar to counter in Java/MapReduce. So you can relate accumulators to counters to understanding it easily
