Test Bench Development

A test bench or testing workbench is a virtual environment used to verify the correctness or soundness of a design or model.

A customized test bench is the best way to approach Python development. Consider a Python test bench as simply an executable .py file that loads all the packages, modules & dependencies and provides a self-sufficient development environment, where users can iteratively develop and test their code.

In the case of pyATS, a successful test bench should include the following components:

  • import all necessary packages & modules

  • testbed devices, and live connections to testbed devices

  • interactive development/prompt

  • iterative code & test, with the final code pluggable into a deliverable

A successful test bench bridges the gap in between knowing the Python language, and designing/writing a first library/testscript in pyATS.

Iterative Development

Everyone can elect to write their own code test bench. Here’s an idea/template that could help you to get started. Keep in mind that the primary goal is to have something where you can code & test what you’ve just written without having to re-run your entire test script, which can be costly when you are loading large number of libraries and connecting to multiple devices.

# Example
# -------
#   example test bench (testbench.py)

#! /bin/env python

# import any modules/libraries/packages & etc
import sys
# ...

# code here
# --------------------------------------------------------------------------
#   develop your test case, library apis & etc here.

# ...

# main block to execute the above test code
# --------------------------------------------------------------------------
if __name__ == '__main__':

    # setup test bench
    # ----------------
    #   setup your work environment, eg:
    #       - load yaml
    #       - connect to devices
    #       - require necessary tcl packages
    #       - etc

    # ...

    # go interactive
    # --------------
    #   note that there's a few ways to go into interactive:
    #       - using IPython
    #       - using pdb.set_trace()
    #       - using code.interact()

    # uncomment/pick one of these:
    from IPython import embed; embed()
    # import code; code.interact(local = locals())
    # import pdb; pdb.set_trace()
    # import ipdb; ipdb.set_trace()

    # cleanup work bench
    # ------------------
    #   do any necessary cleaning, such as:
    #       - remove lingering configurations
    #       - disconnecting from tb devices
    #       - etc

    # ...

    # exit script
    # -----------

    # any code from here onwards is just for your copy/paste purposes
    # ---------------------------------------------------------------
    #   use this space for anything you'd do to copy/paste into your
    #   interactive prompt.

    # ...

When run, the above test bench code stops after setup and present an interactive prompt (assuming working with testbench.py). From there onwards, you can code in the editor, and copy/paste code into the interactive shell and run it to get quick results.

bash$ python testbench.py

Python 3.8.2 (default, Apr  8 2020, 11:06:18)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: sys
Out[1]: <module 'sys' (built-in)>

In [2]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:def my_test_proc():
:    print('this is my test proc working')

In [3]: my_test_proc()
this is my test proc working

In [4]:


using IPython (http://ipython.org/) as an example. IPython is available on PyPI and is the most popular interactive python shell.


IPython %cpaste functionality copes with copy/paste much better than the built-in interactive shell.

Test Bench Example

Here’s a example of writing and executing an aetest based testcase on the fly while maintaing a live device connection without disconnecting.

# Example
# -------
#   aetest test bench with device connections

# import what's necessary
import sys
from pyats import aetest, topology
from genie.utils import Dq

# write a testcase here
# --------------------------------------------------------------------------

class Testcase(aetest.Testcase):
    a testcase written in a test bench environment

    def setup(self, uut, intf = None):

        if intf is None:
            # default to picking the first available interface
            intf = next(iter(uut.interfaces.values()))

        self.intf = intf

    def check_version_string_has_iosxe(self, uut):
        version = uut.execute('show version')
        assert 'IOS XE' in version

    def interface_can_be_configured(self, uut):
        # interface is assumed down, configure it
            interface %s
                ip address
                no shutdown
        ''' % self.intf.name)

        # get the interface information from router_show
        result = uut.parse('show interfaces %s' % self.intf.name)

        # check that the status is reflected
        assert Dq(result).get_values('enabled') == [True]
        assert Dq(result).get_values('ip') == ['']

    def hostname_can_be_changed(self, uut):
        # configure hostname
        uut.configure('hostname %s' % uut.name)

        config = uut.execute('show running-config | include hostname')

        assert uut.name in config

# run test code
# --------------------------------------------------------------------------
if __name__ == '__main__':

    # load testbed from yaml file &
    # grab a device by its alias (uut) for testing
    testbed = topology.loader.load('/path/to/testbed.yaml')
    uut = testbed.devices['uut']

    # connect to the device

    # go interactive, do whatever
    from IPython import embed; embed()

    # exit

    # copy paste code from here onwards..
    # -----------------------------------

    # create a testcase instance from the above code
    tc = Testcase()

    # run the testcase by calling it and providing it with
    # necessary parameters (in this case uut)
    # the result is printed to screen
    tc(uut = uut)

    # --------------------- BACKUP CODE ---------------------

    # Manually Creating Objects
    # -------------------------
    # note that alternatively, instead of loading from YAML, you may also
    # elect to create device and interface objects manually. this avoids the
    # usage of a testbed input file.

    # create a uut device
    uut = topology.Device(name = 'iol',
                          type = 'iol',
                          connections = {
                             'a': {'protocol': 'telnet',
                                               'ip': '',
                                               'port': 10000}
                          credentials = {
                              'default' : {'username': 'admin',
                                           'password': 'adminpw'},
                              'enable' : {'password': 'lab'},

    # add an interface to it
    uut.add_interface(topology.Interface(name = 'Ethernet1/1',
                                         type = 'Ethernet'))

Here’s what the shortest YAML testbed file would look like (with a single interface):

# Example
# -------
#   testbench YAML testbed file

    device_name:                            # use actual device name here
        type: "device_type"
        alias: "alias of device"            # optional
          a:                                # define a single connection
            protocol: "telnet"
            port: 8888

    device_name:                            # use actual device name here
            Ethernet1/1:                    # define a single interface
                alias: "test_intf"          # optional
                type: "ethernet"

The execution of the above code looks like this:

(pyats) [tony@jarvis pyats]$ python testbench.py

... initial connection output ...

Python 3.8.2 (default, Apr  8 2020, 11:06:18)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: tc = Testcase()

In [2]: tc()

... output of the actual testing ...

Out[2]: Passed

In [3]:

And … the testcase passed.

Notice that within this bench environment, you can simply instantiate a Testcase object tc = Testcase(), and run it by calling that instance tc() and passing it any testcase parameters it requires. This, combined with %cpaste copy/pasting functionality, enables users to get immediate feedback of the code (in this case, a Testcase) that’s just been written, by simply treating them as callable class instances.


this is a method of development. the concept also extends to other test script sections, as well as the development of libraries & packages.


use the python debugger (pdb) when encountering issues.


use Ctrl-D to exit