apache-spark Text files and operations in Scala Join two files read with textFile()

Help us to keep this website almost Ad Free! It takes only 10 seconds of your time:
> Step 1: Go view our video on YouTube: EF Core Bulk Insert
> Step 2: And Like the video. BONUS: You can also share it!

Example

Joins in Spark:

  • Read textFile 1

    val txt1=sc.textFile(path="/path/to/input/file1") 
    

    Eg:

      A B
      1 2
      3 4
    
  • Read textFile 2

    val txt2=sc.textFile(path="/path/to/input/file2") 
    

    Eg:

      A C
      1 5
      3 6
    
  • Join and print the result.

    txt1.join(txt2).foreach(println)
    

    Eg:

      A B C
      1 2 5
      3 4 6
    

The join above is based on the first column.



Got any apache-spark Question?