pyats.async package

Module:

async_

Authors:

Siming Yuan (siyuan), CSG Test - Ottawa

Submodules

exception pyats.async_.exceptions.ChildProcessException

Bases: Exception

class pyats.async_.exceptions.ExceptionWithTraceback(exc)

Bases: object

exception pyats.async_.exceptions.RemoteException(tb)

Bases: Exception

pyats.async_.exceptions.filter_exception(tb)
pyats.async_.exceptions.rebuild_exc(exc, tb)
class pyats.async_.parallelcall.ChildLabor(index, queue, *args, **kwargs)

Bases: multiprocessing.context.ForkProcess

ChildLabor Class

Worker class, subclassing from multiprocessing.Process under ‘fork’ context. The customizations in this class is designed purely for it to be called under p-call.

built-in init

on top of native multiprocessing.Process instanciation, add also the index (order) and queue (result pipe)

Parameters

  • (int) (index) –

  • (queue) (queue) – parent.

run()

Runs the given target function in child processes. This is the default entry point of multiprocessing.Process

Updates the Steps object with an offset if there is one.

Returns

  • nothing. However, all returns and errors arising from the called

  • user target will be stored into the pipeline (queue) in the format

  • (index, boolean, result)

  • where the boolean represents whether the func call was successful (

  • no exception), and the result being either the exception, or the

  • result of the call

class pyats.async_.parallelcall.ChildWork(target, args, kwargs)

Bases: tuple

Create new instance of ChildWork(target, args, kwargs)

args

Alias for field number 1

kwargs

Alias for field number 2

target

Alias for field number 0

class pyats.async_.parallelcall.Pcall(targets, timeout=None, cargs=(), iargs=(), ckwargs={}, ikwargs=(), **varkwargs)

Bases: object

Pcall class

allows users to perform and control parallel calls (to an extend) using this class. Initialization of this class allows for various combination of arguments to be zipped together and passed to each parallel called instance.

Parameters

  • (list/callable) (targets) –

  • (int) (timeout) –

  • (tuple/list) (ikwargs) –

  • (tuple/list) – each instance. Each item should be also a tuple/list

  • (dict) (ckwargs) –

  • (tuple/list) –

  • (dict of list/tuple) (varkwargs) – value is a list of values corresponding to that key-value argument of an instance.

Rules:

  1. the number of parallel instances depends on the following factor:

    1. number of targets provided (if more than 1)

    2. number of instance arguments (iargs or ikwargs)

    3. number of instance values (varkwargs) combined together, whichever is shorter.

  2. the order of arguments is positional sensitive

  3. the order of results list is the same as the child process order

join()

blocks the current process until timeout is reached, or if all childs finish executing their given job. Collects & sets the result variable after everything is finished. If a child process raises an exception during execution, the same exception will be thrown here.

kill()

kill (SIGKILL) all child processes.

property livings

livings

returns the list of all processes that are currently alive.

Type

property

classmethod pcall(*args, **kwargs)

pcall classmethod

Classmethod to allow short-cut calling of Pcall. This api wraps Pcall’s start(), join() apis together, blocks and returns the results of parallel calls in the order they were called.

property pids

list of pids after it’s all started

start()

starts all child worker processes

terminate()

terminates all child processes.

Note

this might take a split second before it taking effect

class pyats.async_.synchronize.Lockable(*, lock=<bound method BaseContext.RLock of <multiprocessing.context.ForkContext object>>)

Bases: object

Base class that adds locking/unlocking (semaphore) to a class object, allowing the class to be called in multiprocessing & threading without deadlocks: all method calls should first acquire the lock, perform its dedicated action, then release the lock.

Subclassing this base-class is the preferred way to go when creating classes that support locking and unlocking.

Note

  • user shall explicitly call this class’ __init__ using super()

  • user shall use this class’s lock() method instead of the generic one decorating the methods that needs to be locked/unlocked.

__init__

api to initialize the base class and set the built-in lock.

Parameters

(multiprocessing.Lock) (lock) – must be a subclass of mp.Lock().

Note

Beware that changing the lock can have detremental effects, eg, Lock() vs RLock() differences, etc.

acquire(block=True, timeout=None)

acquire the lock on this class instance. See python multiprocessing documentation for details on lock acquire.

Parameters

  • (bool) (block) – acquired by this process/thread. Default to True,

  • (float) (timeout) – many seconds before returning.

Returns

Return type

True/False for whether lock was acquired or not

static locked(method)

locked decorator

This is the lock decorator defined exclusively for the Lockable() class. It does the exact same thing as the generic lock() decorator, but eliminates the need to explicitly check for whether locking object exists or not.

Parameters

(func) (method) –

Returns

Return type

new wrapped method that automatically invokes the lock

release()

releases the lock on this class instance. See python multiprocessing documentation for details on lock release.

Only call this method when the calling process or thread owns the lock. An AssertionError is raised if this method is called by a process or thread other than the owner or if the lock is in an unlocked (unowned) state.

class pyats.async_.synchronize.LockableMeta(name, bases, attrs)

Bases: type

Metaclass that adds locking/unlocking (semaphore) to a class’s methods, allowing the class to be called in multiprocessing and multithreading without deadlocks: all method calls will first acquire the lock, perform its dedicated action, then release the lock.

The use of this metaclass eliminates user’s need to explicitly use lock() decorator on each API.

The limitation of this metaclass is simple: if the user doesn’t make any lock() api calls before forking, all locks fail.

Metaclass __new__

Metaclass api called before class creation. Hijacks the class creation process and add to all function calls, a locking/unlocking semaphore for multiprocessing scenarios for avoiding deadlocks.

Parameters

  • (str) (name) –

  • (list) (bases) –

  • (dict) (attrs) –

static ignore(obj)

decorator that allows user to mark an object (method) to be ignored by the automatic mechanism that flags all methods as lock() methods. This allows certain helper APIs to be called without waiting for a lock on the parent instance.

Parameters

(obj) (obj) –

pyats.async_.synchronize.locked(method)

lock decorator

This is a generic decorator method that adds locking (semaphore) to a class method, allowing the class to posess locking/unlocking capabilities in a forked multiprocessing & multi-threading scenario.

Parameters

(func) (method) –

Returns

Return type

new wrapped method that automatically invokes RLock()

Note

internal implementation based on adding ‘__lock__’ attribute to parent class instance, and using multiprocessing.Rlock() class.

due to the above, if ‘__lock__’ is instantiated first in a child process, the locking/unlocking mechanism would be broken. Make sure to always instantiate the lock (by calling an lock function or equivalent) inside the main process before forking.