Jobs were introduced in PowerShell 2.0 and helped to solve a problem inherent in the command-line tools. In a nutshell, if you start a long running task, your prompt is unavailable until the task finishes. As an example of a long running task, think of this simple PowerShell command:
Get-ChildItem -Path c:\ -Recurse
It will take a while to fetch full directory list of your C: drive. If you run it as Job then the console will get the control back and you can capture the result later on.
This document will show you how to call Scala jobs from a pyspark application.
This approach can be useful when the Python API is missing some existing features from the Scala API or even to cope with performance issues using python.
In some use cases, using Python is inevitable e.g you are building models with