Python Language Python concurrency Passing data between multiprocessing processes
Example
Because data is sensitive when dealt with between two threads (think concurrent read and concurrent write can conflict with one another, causing race conditions), a set of unique objects were made in order to facilitate the passing of data back and forth between threads. Any truly atomic operation can be used between threads, but it is always safe to stick with Queue.
import multiprocessing
import queue
my_Queue=multiprocessing.Queue()
#Creates a queue with an undefined maximum size
#this can be dangerous as the queue becomes increasingly large
#it will take a long time to copy data to/from each read/write thread
Most people will suggest that when using queue, to always place the queue data in a try: except: block instead of using empty. However, for applications where it does not matter if you skip a scan cycle (data can be placed in the queue while it is flipping states from queue.Empty==True to queue.Empty==False) it is usually better to place read and write access in what I call an Iftry block, because an 'if' statement is technically more performant than catching the exception.
import multiprocessing
import queue
'''Import necessary Python standard libraries, multiprocessing for classes and queue for the queue exceptions it provides'''
def Queue_Iftry_Get(get_queue, default=None, use_default=False, func=None, use_func=False):
'''This global method for the Iftry block is provided for it's reuse and
standard functionality, the if also saves on performance as opposed to catching
the exception, which is expencive.
It also allows the user to specify a function for the outgoing data to use,
and a default value to return if the function cannot return the value from the queue'''
if get_queue.empty():
if use_default:
return default
else:
try:
value = get_queue.get_nowait()
except queue.Empty:
if use_default:
return default
else:
if use_func:
return func(value)
else:
return value
def Queue_Iftry_Put(put_queue, value):
'''This global method for the Iftry block is provided because of its reuse
and
standard functionality, the If also saves on performance as opposed to catching
the exception, which is expensive.
Return True if placing value in the queue was successful. Otherwise, false'''
if put_queue.full():
return False
else:
try:
put_queue.put_nowait(value)
except queue.Full:
return False
else:
return True
