Jobs were introduced in PowerShell 2.0 and helped to solve a problem inherent in the command-line tools. In a nutshell, if you start a long running task, your prompt is unavailable until the task finishes. As an example of a long running task, think of this simple PowerShell command:
Get-ChildItem -Path c:\ -Recurse
It will take a while to fetch full directory list of your C: drive. If you run it as Job then the console will get the control back and you can capture the result later on.
Active Job is a framework for declaring jobs and making them run on a variety of queuing backends. These jobs can be everything from regularly scheduled clean-ups, to billing charges, to mailings. Anything that can be chopped up into small units of work and run in parallel, really.
This document will show you how to call Scala jobs from a pyspark application.
This approach can be useful when the Python API is missing some existing features from the Scala API or even to cope with performance issues using python.
In some use cases, using Python is inevitable e.g you are building models with