pyats.kleenex package¶

Module:

kleenex

Authors:

Myles Dear(mdear), CSG Test - Ottawa Siming Yuan (siyuan), CSG Test - Ottawa

Description:

This module provides a framework that allows users to:

plug in own orchestrators that bring up dynamic device topologies on a variety of different backends.
plug in their own cleaners that prepare a physical device for testing.

Subpackages¶

Submodules¶

This module contains the pyATS bringup base classes from which all bringup implementations must inherit.

class pyats.bringup.bases.BringUpBase(*args, **kwargs)¶

Bases: object

Base class for a dynamic topology orchestrator.

clean_config¶

Dictionary representation of the full clean configuration, if specified by the user.

Type: dict

bringup_clean_config¶

Dictionary representation of the subset of the clean configuration that specifies bringup worker configuration, if specified by the user.

Type: dict

autogenerated_clean_config¶

Dictionary representation of the autogenerated clean configuration, if populated by the bringup worker. All device names are actual and not logical. This configuration details the inputs used by a “mini-clean” done within the scope of bringup before the newly brought up topology is handed back for possible post-bringup cleaning. This “mini-clean” is often used to “babysit” virtual devices until they are up and ready for testing.

Type: dict

output_clean_config¶

Dictionary representation of the output clean configuration, if populated by the bringup worker, to be merged on top of the user-specified clean configuration.

Type: dict

logical_devices¶

Devices in the logical testbed config yaml file (if specified by the user) that are declared to be logical by use of logical: True.

Type: list

Create an instance of a dynamic topology orchestrator.

autogenerated_clean_config = None¶

bringup_clean_config = None¶

clean_config = None¶

launch_topology()¶: Launch the topology, wait for it to come up, leave it running.

logical_devices = []¶

output_clean_config = None¶

tear_down_topology()¶: Destroy the topology and reclaim all associated resources.

Note

Orchestrators (bringup workers) run in a worker subprocess.

Please ensure that you make a call to pyats.bringup.manager.request_bringup_worker_server_shutdown when you are done tearing down your topology. This ensures that the bringup worker subprocess gracefully terminates at the appropriate time.

The bringup worker is expected to install the appropriate signal handlers to guarantee all resources are properly cleaned up if the worker subprocess gets interrupted with a signal.

class pyats.bringup.bases.BringUpWorkerBase(*args, topology_name=None, interactive=False, easypy=False, taskid=None, runinfo_dir=None, testbed_file=None, logical_testbed_file=None, clean_config=None, bringup_clean_config=None, bringup_log_level=None, skip_teardown=False, tb_yaml_output_file_name=None, **kwargs)¶

Bases: pyats.bringup.bases.BringUpBase

Base class to be inherited by bringup worker (orchestrator) classes.

This class provides common infrastructure that any bringup implementation would need, including:

asyncio framework (including exception handling model).
merge an input (sparse) testbed config with the raw testbed config coming from the orchestrator.
logical to actual device and interface name matching
ability to pull in pluggable CLI help content from the inheriting worker class. This allows a consolidated help display to be provided to the decoupled bringup tool.

logical_virtual_devices¶

Devices in the testbed config yaml file (if specified by the user) that are declared to be logical and virtual by use of logical: True and device: type specifying a supported virtual device type.

Type: list

static_testbed_config¶

If the user specifies a testbed YAML, this variable contains the configuration details specified in that file.

Type: dict

logical_testbed_config¶

If the user specifies a logical testbed YAML, this variable contains the configuration details specified in that file. It is expected that if this file specifies logical virtual devices by containing logical: True and having a device: type declared whereby the type is a member of allowed_virtual_device_types. This file is considered sparse and not able to be validated until testbed configuration generated by the orchestration backend is merged in, the dynamic router placeholder sections are removed and logical-to-actual mapping is done.

Type: dict

Create an instance of a dynamic topology orchestrator.

Users inheriting from this class are expected to call super().__init__ (*args, **kwargs) as their very first line.

Parameters

topology_name (str) – A name for this topology. If specified, a timestamp is appended to it to ensure uniqueness.
interactive (bool) – Whether or not the testbed is being brought up in a standalone interactive tool. Internal parameter (not set via CLI). Defaults to False.
easypy (bool) – If True, then the testbed is being launched from an easypy plugin. Internal parameter (not set via CLI). Defaults to False.
taskid (str) – The unique taskid, used when launched via an easypy plugin. Internal parameter (not set via CLI). Defaults to None.
runinfo_dir (str) – The directory to place orchestrator logs and the synthesized testbed YAML file into (passed in when invoked via easypy). Internal parameter (not set via CLI). Defaults to None.
skip_teardown (bool) – If True, and easypy is True, then do not automatically tear down the topology when so requested and warn the user they are now responsible for tearing it down. If False, then tear down the topology when so requested. If easypy is False this option is ignored. Defaults to False.
testbed_file (str) – The testbed config YAML filename (passed in when invoked via easypy). This testbed config may contain only static (non-logical) devices. If not specified, the testbed file is taken from the CLI input parameter -testbed_file. Defaults to None.
logical_testbed_file (str) – The testbed config YAML filename (passed in when invoked via easypy). This testbed config may contain either static (non-logical) or logical devices. If not specified, the testbed file is taken from the CLI input parameter -logical_testbed_file. Defaults to None.
bringup_log_level (str) – Logging level to use for bringup. One of the following : debug, info, warning, error, critical. If not specified, the value from bringup/log_level in the clean configuration YAML is used. Defaults to info.
tb_yaml_output_file_name (str) – The name of the synthesized pyATS testbed YAML file that is automatically written after the topology has been launched. If not specified but easypy is True, a unique name is chosen to be written to the runinfo directory. Defaults to None, meaning no file is written.
clean_config (dict) – A dictionary holding user-specified clean configuration content. Orchestrator configuration content is a subset of the clean configuration. This parameter is internal and cannot be specified via CLI. Defaults to {}.
bringup_clean_config (dict) – A dictionary holding the subset of the user-specified clean configuration content under the orchestrator class. This parameter is internal and cannot be specified via CLI. Defaults to {}.

dev_name_xref¶

This dict must be populated by the inheriting class. It is indexed by actual device name and contains the corresponding logical device name.

Type: dict

dev_intf_xref¶

This dict is populated as part of the just-in-time testbed topology processing logic. It is indexed by actual device name and actual interface name and contains the corresponding logical interface name.

Type: dict

event_loop¶

Event loop for running asyncio tasks. This loop is used to coordinate all the async events occuring during testbed launch and teardown.

Type: asyncio.unix_events._UnixSelectorEventLoop

help¶

This gets set to true when the user specifies -help via the command line (as is the case when they type bringup -help).

Type: bool

shutting_down¶

Set to True when the topology is shutting down. This is used by the signal handling model to ensure the topology is not torn down if it is already shutting down.

Type: bool

testbed_config¶

Contains the testbed YAML content emitted by the orchestrator using information only available at run-time (such as console connection details). This content is merged with logical_testbed_config and static_testbed_config (if specified) and saved in this variable. This testbed config is meant to be loaded into a pyATS topology object.

Type: dict

topology_name¶

A unique name for this topology.

Type: str

logical_devices¶

A list of logical devices in the provided topology.

Type: list

logical_virtual_devices¶

A list of logical virtual devices in the provided topology. A virtual device is one whose type is one of the allowed virtual device types.

Type: list

logical_physical_devices¶

A list of logical physical devices in the provided topology.

Type: list

actual_devices¶

A list of actual devices populated by the orchestrator.

Type: list

classmethod authenticate(*args, **kwargs)¶: Authenticate the user. This method is called by the bringup manager server to ensure the user is authenticated. It must be implemented by the inheriting class as it might not be required by all workers.

launch_topology()¶: Schedule a virtual topology to be launched.

logical_physical_devices = []¶

logical_testbed_config = {}¶

logical_virtual_devices = []¶

set_interactive(interactive)¶: Setter to allow member to be set by xmlrpc client.

set_skip_teardown(skip)¶: Setter for skip_teardown

static_testbed_config = {}¶

tear_down_topology(raise_stored_exception=True)¶: Schedule a virtual topology to be gracefully torn down.

update_help()¶

Update help from my CLI parser.

Implementors inheriting from this class are expected to first call this parent implementation, then update the help from their own parser.

A CLI parser for BringUpWorkerBase

pyats.bringup.base_cli_parser.populate_base_bringup_parser(parser)¶: Add bringup options to a parser.

Module responsible for bringing up and tearing down dynamic topologies in their own subprocesses, to ensure graceful teardown always occurs.

class pyats.bringup.manager.BringUp(*args, orchestrator=None, clean_files=None, bringup_log_level=None, clean_config=None, bringup_clean_config=None, debug=False, **kwargs)¶

Bases: pyats.bringup.bases.BringUpBase

Bring up and tear down a dynamic topology.

Parameters may be specified either directly or via the command line.

This class causes the bringup worker object to be instantiated in a subprocess and manages it thereafter via a proxy.

Instantiate a bringup proxy instance.

This object follows the same base API as a bringup worker, but causes the actual worker object to be instantiated in a subprocess.

Requests to this object are proxied to the worker object.

Parameters

clean_files (list) – The loadables containing the clean configuration dict or files containing YAML text.
orchestrator (str) – The bringup worker class to instantiate. If not specified, the value from bringup/class in the clean configuration YAML is used. Otherwise, an AssertionError is thrown.
bringup_log_level (str) – Logging level to use for bringup. One of the following : debug, info, warning, error, critical. If not specified, the value from bringup/log_level in the clean configuration YAML is used. Defaults to info.
clean_config (dict) – Clean configuration (which contains the bringup worker configuration) if not specified via clean_file. This parameter is internal and cannot be specified via CLI.
bringup_clean_config (dict) – Bringup worker configuration (which is a subset of clean configuration) if not specified implicitly via clean_file. This parameter is internal and cannot be specified via CLI.
debug (bool) – Whether or not to enable debug mode. If specified as True and a bringup worker fails to instantiate or throws an exception an interactive debugger is launched at the point of failure. It also becomes possible for a user to insert pdb.set_trace() calls in their worker and perform debugging on the worker as it runs in its forked subprocess. If specified as False the worker cannot be debugged. Defaults to False. This parameter is internal and cannot be specified via CLI.

bringup_worker¶

This object is created when the user enters a context manager block and represents by proxy the actual bringup instance running in a forked subprocess.

Type: xmlrpc.client.ServerProxy

manager_cli_help_details¶

CLI parser help details populated by pyats.kleenex.bases.BringUpWorkerBase.update_help. Used to ensure a consistent display when an orchestrator’s xxxxbringup -help is called.

Type: list

manager_cli_usage_details¶

CLI parser usage details populated by pyats.kleenex.bases.BringUpWorkerBase.update_help. Used to ensure a consistent display when bringup -help is called.

Type: list

Raises: AssertionError – If the bringup worker class is not specified.

property cli_help_details¶: Retrieve the CLI help details for the bringup -help command.

property cli_usage_details¶: Retrieve the CLI usage details for the bringup -help command.

property help¶: Retrieve whether or not the topo was launched in help mode.

property interactive¶: Retrieve whether or not the topo was launched in interactive mode.

launch_topology()¶: Command the bringup worker to launch the topology.

property logical_devices¶: Retrieve list of logical devices.

property output_clean_config¶: Retrieve the output clean configuration. The user may use this property to merge just-in-time clean configuration on top of static clean configuration originating from the clean YAML file.

set_skip_teardown(value=True)¶: Command the bringup worker to skip teardown.

tear_down_topology()¶: Command the bringup worker to tear down the topology.

property topology_config¶: Retrieve the post-bind testbed topology. The user uses this property to retrieve the testbed configuration when the orchestrator has successfully bound the logical topology to an actual topology.

property topology_name¶: Retrieve the topology name. The user uses this property to retrieve the topology name.

update_help()¶: Update help from my CLI parser.

pyats.bringup.manager.request_bringup_worker_server_shutdown()¶: Request XMLRPC server shutdown. The last thing the bringup worker must do before returning after tearing down the topology is to request its server to shut itself down. This is necessary to return control back to the calling user.

A CLI parser for BringUp

pyats.bringup.manager_cli_parser.populate_bringup_manager_parser(parser)¶: Add bringup manager options to a parser.

Signal handling to ensure graceful topology termination.

pyats.bringup.signals.install_signal_handlers()¶: Set up handlers for all possible signals to ensure the virtual testbed gets taken down correctly.

class pyats.kleenex.engine.KleenexEngine(reporter, debug_=False, **clean_mappings)¶

Bases: object

KleenexEngine effectively allows users to clean multiple devices at the same time, using various clean implementations for each device, without having to worry about about forks, processes & etc.

Internally this class is just about the same as async_.pcall, however, as clean definitions are designed to be mostly reused, there are some overlaps in code…

clean(devices, timeout=None, task=None, lcl_logger=<Logger pyats.kleenex.engine (WARNING)>)¶

Clean one or more sets of devices. Devices within each set are cleaned in parallel (if not running in debug mode).

Each set is cleaned sequentially.

clean_testbed(testbed, *, timeout=None, task=None, lcl_logger=<Logger pyats.kleenex.engine (WARNING)>)¶: Performs best-attempt cleaning of the testbed, assuming there is clean definition for each device in this testbed. If none is found - that device is skipped

classmethod from_cleandefs(*cleandefs, reporter, debug_=False)¶

has_cleandef(device)¶

Test whether KleenexEngine has clean definitions for a given device.

Parameters: device (device object or device name string) –
Returns
Return type: True or False

static update_testbed(testbed, **cleaninfo)¶

exception pyats.kleenex.exceptions.CleanException(tb)¶: Bases: Exception

class pyats.kleenex.exceptions.CleanExceptionWithTraceback(exc, device_name)¶: Bases: pyats.clean.exceptions.ExceptionWithTraceback

exception pyats.kleenex.exceptions.CleanWorkerException¶: Bases: Exception

A decoupled tool to perform bringup and/or clean. If bringup is selected, launch the topology, optionally clean the topology, wait for the user to press <Ctrl><C>, then tear down the topology.

returns

Returns a Linux exit code of 0 if all actions succeeded.
Otherwise, returns a Linux exit code if 1 if one or more actions failed.

class pyats.kleenex.kleenex_main.KleenexMain(testbed_file=None, clean_files=None, clean_devices=None, loglevel=20, logdir=None, no_logdir=None, orchestrator=None, debug=False, no_mail=False, commandline=False, device_image_deprecated=None, os_image_deprecated=None, clean_separator=':', device_image=None, os_image=None, group_image=None, platform_image=None, legacy=True, parser=None, no_upload=False, runinfo_dir=None, archive_dir=None, no_archive_subdir=False, skip_teardown=False, **kwargs)¶

Bases: object

This class implements bringup/clean behavior.

Bringup describes how a logical topology is converted into an actual topology consisting of physical and/or software-based routers.

Clean describes how one or more physical devices are prepared for testing by placing user-specified images on them. It may also include such steps as netbooting, bringing up a device from rommon mode or power cycling a device in order to recover it.

Create an instance of a bringup/clean orchestrator.

If an exception is thrown or a CLI argument fails validation, sys.exit(1) is called. This allows proper integration with test runners such as Jenkins.

Parameters

testbed_file (str) – A YAML-formatted file describing the devices in the topology and how they are interconnected.
clean_files (list) – YAML-formatted files describing how the devices in the topology are to be brought up and / or cleaned.
clean_devices (list) – A list of devices to clean. If not specified, all devices in the topology are cleaned.
loglevel (str) – The Kleenex log level. One of the following : DEBUG, INFO, WARNING, ERROR, CRITICAL.
runinfo_dir (str) – The directory to store Kleenex logs and console output to. If not specified, defaults to a time-stamped directory under runinfo directory in the current environment, with a prefix of “Kleenex”.
archive_dir (str) – The directory to store Kleenex archive logs (zip). If not specified, defaults to a date-stamped directory under archive directory in the current environment.
no_archive_subdir (bool) – When True, does not create a date-stamped subdirectory under the specified archive directory. The Kleenex archive will be saved to the archive directory directly.
no_logdir (bool) – If True, no log directory is produced and all output goes to the console. If False, a log directory is produced as described by the logdir parameter. Defaults to False.
orchestrator (pyats.kleenex.bases.BringUpWorkerBase) – The bringup orchestrator to use. If not specified, then bringup is skipped.
skip_teardown (bool) – If True, topology is not torn down at the end of the run If False , topology is torn down at the end. Defaults to False.
no_mail (bool) – If True, no email is sent on failure. If False, an email is sent on failure Defaults to False.
commandline (bool) –
If True, the object is being instantiated as part of a standalone tool and all arguments appear in the -help display.

If False, the object is being instantiated programmatically and the user is expected to pass arguments to the constructor instead of via the command line. Defaults to False.
legacy (bool) –
If True, the commandLine arguments use the legacy version such as -testbed_file

If False, the commandLine arguments use the current version such as --testbed-file
parser (obj) – Provide a custom parser object
no_upload (bool) – If True, log archive uploading is disabled. If False, log archives are uploaded to TaaS Defaults to False.

parser¶

CLI parser that does help processing and contains help/usage details from other CLI parsers contained by the module.

Type: pyats.kleenex.utils.ArgvQuotingParser

local_parser¶

CLI parser for local module-scope arguments.

Type: pyats.kleenex.utils.ArgvQuotingParser

bringup¶

Bringup instance containing usage and help text.

Type: pyats.kleenex.bringup_manager.BringUp

help_requested¶

Set to True then this class has been instantiated in commandline mode and the user has specified -h on the command line. Otherwise, defaults to False.

Type: bool

cleanup()¶

configure_local_parser(commandline, legacy)¶: Create parser for all local module-scope arguments except help.

configure_logging()¶

configure logging Configure logging level based on the values from CLI args parsing and values passed to the method.

Parameters: None –

configure_parser(commandline, legacy)¶: Create a parser integrating help content with contained parsers.

do_clean_if_required()¶

generate_archive_file()¶: Generate archive file for clean

generate_task_email_reports()¶

get_results_zipfile_content()¶

Produce zipfile content for all results.

Returns
Return type: Binary representation of the zipfile content.

instantiate_bringup_worker_if_required()¶

launch_dynamic_topology_if_required()¶: Lauch a dynamic topology if logical devices have been specified.

run(**kwargs)¶: Execute bringup and / or clean behavior as required.

send_detailed_clean_email_report_to_user(resp=None, msg=None)¶: In case of error, send email to user with details.

start_liveview()¶

stop_liveview()¶

teardown_dynamic_topology_if_required()¶

update_contained_help()¶

Consolidate a parser’s usage and help text from contained objs.

Updates the help output of a provided parser. Automatically aggregates its current help information with those of the plugins. this allows an orchestrator’s xxxbringup -help to show all available arguments from every class along the chain.

upload_results_to_taas()¶

upload_results_to_xpresso_if_required()¶

write_env_file()¶

Utilities to filter and format exceptions.

This module allows stack traces to be simplified for user consumption, ensuring parts of the traceback that are either internal to the pyATS core or otherwise considered noisy, confusing or not adding value for support purposes to be omitted from the final customer-facing exception output.

The format_exception API from the core traceback package is overloaded here and given a new parameter, skip_pred.

A skip predicate is evaluated for each stack frame in an exception, and in all associated cause/context stack frames as well. If the predicate evaluates to False, it is not included in the traceback output.

pyats.clean.traceback.format_exception(etype, value, tb, limit=None, chain=True, skip_pred=None)¶

Format a stack trace and the exception information.

The arguments have the same meaning as the corresponding arguments to print_exception(). The return value is a list of strings, each ending in a newline and some containing internal newlines. When these lines are concatenated and printed, exactly the same text is printed as does print_exception().

This module contains the Kleenex YAML schema used by clean and bringup.

pyats.clean.schema.allowed_virtual_device_types¶

The list of allowed virtual device types.

Type: list of str

This module contains common utilities required by bringup.

pyats.clean.utils.help_suppress_kleenex(msg)¶

Suppress this argument if called as a decoupled kleenex tool. Since kleenex requires clean and testbed YAML files to be specified, all DE-scope arguments are to be suppressed. A DE-scope argument always has an equivalent in the clean YAML schema. The DE-scope arguments may be accessed via the normal orchestrator- specific bringup tool.

Parameters: msg (str) – The help text that is suppressed in the kleenex -help display and otherwise appears in the decoupled bringup tool’s xxxbringup -help display.

pyats.clean.utils.parse_cli_args(parser)¶

Parse known CLI arguments found in sys.argv. After parsing the known arguments remove them from sys.argv and keep the rest.

Implementation copied from pyats.aetest.main.parseArgs.

Returns
Return type: dictionary of argument name/value where value is not None

pyats.clean.utils.tear_down_all_topologies()¶: Tear down all topologies. This is typically called when an exception is caught after topologies have already been brought up.

pyats.clean.utils.testbed_config_contains_logical_routers(testbed_config)¶

testbed_config: file or dict

The loadable containing testbed configuration details as passed in by the user via the testbed_config CLI parameter. The loadable may be in one of the following forms:

dictionary respecting the yaml schema format

full path/name of the yaml testbed file

any yaml python stream readable

Returns: True if one or more routers in the testbed config YAML file contain the key logical: True, False otherwise.
Return type: bool

pyats.clean.utils.validate_clean_devices(clean_devices)¶

Validate clean devices.

Returns

None if no devices to clean.
Otherwise, the devices to clean in a two-deep list format
[[‘dev1’, [‘dev2’, ‘dev3’], [‘dev4’]]
This input causes the following actions to be taken (preceding
step must be complete for the next step to start) –
1. dev1 is cleaned.
2. dev2/dev3 are cleaned in parallel.
3. dev4 is cleaned
A single-deep list format is also supported. In the following
example, dev1, dev2, dev3 and dev4 are cleaned in parallel
[‘dev1’, ‘dev2’, ‘dev3’, ‘dev4’]

The following formats are allowed:

A string containing a space-separated list of devices. devices specified in quotes are made part of the list. All devices in this list are cleaned in parallel. dev1 dev2 dev3 “dev4 dev5”
A YAML-formatted string “[dev1, [dev2, dev3], dev4]”

Validation fails if the user nests the lists more than one level deep, for example : [a, b, [c, d, [e, f]]]
A list of device names.

class pyats.kleenex.worker.KleenexSteps(parent=None)¶

Bases: pyats.aetest.steps.implementation.Steps

built-in __init__

instantiates the Steps class by creating an internal list that houses number of steps currently taken.

Note

step index is the same as the list index of current step.

start()¶

Creates a new Step instance and return it to the user scope. This is the main API called within the user realm to create and start new steps within the current scope.

Offset is used primarily with parallel call implementation. If specifying manually, ensure offset is large enough to enable additional steps per chid process.

Parameters

(int) (offset) –
args (all arguments to this api propagates directly to Step) –
kwargs (all arguments to this api propagates directly to Step) –

Returns

Return type

new Step() instance.

class pyats.kleenex.worker.KleenexWorker(cleaner, device, reporter, logfile, queue, timeout, debug_=False)¶

Bases: multiprocessing.context.ForkProcess

property result¶

run()¶: Method to be run in sub-process; can be overridden in sub-class