Tutorial by Examples

Prototype: aggregate(zeroValue, seqOp, combOp) Description: aggregate() lets you take an RDD and generate a single value that is of a different type than what was stored in the original RDD. Parameters: zeroValue: The initialization value, for your result, in the desired format. seqOp: ...

Spark uses lazy evaluation; that means it will not do any work, unless it really has to. That approach allows us to avoid unnecessary memory usage, thus making us able to work with big data. A transformation is lazy evaluated and the actual work happens, when an action occurs. Example: In [1]: li...

In spark-shell: sc.version Generally in a program: SparkContext.version Using spark-submit: spark-submit --version