apache-spark Calling scala jobs from pyspark How to call spark-submit
Example
To call this code you should create the jar of your scala code. Than you have to call your spark submit like this:
spark-submit --master yarn-client --jars ./my-scala-code.jar --driver-class-path ./my-scala-code.jar main.py
This will allow you to call any kind of scala code that you need in your pySpark jobs
