Section Steps¶
In aetest, testscripts are naturally broken down into TestContainers
and TestFunctions, two-level segments defined in Object Model. For
example:
CommonSetup:subsection
Testcase:setup/test/cleanup
This two-level approach is mostly sufficient when sections can be kept short &
independent. However, often times, there’s a need for larger, longer sections,
mostly due to nature of the tests being carried out. In these situations, users
can leverage the optional section Steps feature: slicing down large
sections into groups of smaller, more granular linear actions that sums up
towards the same testing goal.
Defining Steps¶
To divide sections into smaller steps, define section methods with the
Reserved Parameters steps in its function arguments. This enables
steps creation by bringing Steps class instance into the local method scope.
Each step is started by calling steps.start(), and providing it a short
descriptive name for that new step. A longer description can also be provided,
which will be included in the final report. The returned Step Objects is a
python Context Manager, intended to be consumed by the with statement.
# Example
# -------
#
# breaking a test section into smaller steps
from pyats import aetest
# using a testcase as example
# applies to all sections (subsection/setup/cleanup/test)
class NeilArmstrong(aetest.Testcase):
# define section method with 'steps' reserved parameter argument
# this enables the engine to pass steps object to the local scope
@aetest.test
def says(self, steps):
# breaking down this test into two steps
# using python "with" statement and steps parameter
with steps.start('first step',
description = 'this is the first step'):
print('one small step for [a] man')
with steps.start('second step',
description = 'this is the second step'):
print('one giant leap for mankind')
# script output
#
# +------------------------------------------------------------------------------+
# | Starting section says |
# +------------------------------------------------------------------------------+
# +..............................................................................+
# : STEP 1: first step :
# +..............................................................................+
# one small step for [a] man
# The result of STEP 1: first step is => PASSED
# +..............................................................................+
# : STEP 2: second step :
# +..............................................................................+
# one giant leap for mankind
# The result of STEP 2: second step is => PASSED
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - first step Passed
# STEP 2 - second step Passed
# ------------------------------------------------------------
# The result of section says is => PASSED
The usage of Context Manager allows each step’s code to properly nest under its own indentation block, allows the test infrastructure to identify each step’s boundaries, and allows independent error catching & result handling etc.
In addition, this returned Step Objects can be stored into variable using
as statement, allowing users to access/refer to current step information,
and make calls to result apis:
# Example
# -------
#
# referring to the step object
from pyats import aetest
class Testcase(aetest.Testcase):
@aetest.test
def test(self, steps):
# create step into 'step' variable
with steps.start('the first step') as step:
# eg, each step is automatically assigned a unique,
# numeric index number, and can be accessed using the
# "index" attribute.
print('Current step index: ', step.index)
# Current step index: 1
Step Results¶
Each step comes with its own result, represented using Result Objects. When a script section is segmented using steps feature, that section’s result is calculated using the combined roll-up results of all its contained steps.
All other step result behaviors are identical to its parent Section Results:
the default result for all steps is
Passed
AssertionError: AssertionError exceptions corresponds toFailed
Exception: Exceptions corresponds toErrored
# Example
# -------
#
# step results default
from pyats import aetest
class Testcase(aetest.Testcase):
@aetest.test
def test_one(self, steps):
# default step result -> Passed
with steps.start('the passed step'):
pass
# AssertionErrors -> Failed
with steps.start('the failed step'):
assert 1 == 0
@aetest.test
def test_two(self, steps):
# general Exceptions -> Errored
with steps.start('the errored step'):
# generate a python error
{}['non existent key']
# script output
# note that the result of each test section is the
# combined roll-up of its steps. Eg:
# test_one: Passed + Failed -> Failed
# test_two: Errored -> Errored
#
# +------------------------------------------------------------------------------+
# | Starting testcase Testcase |
# +------------------------------------------------------------------------------+
# +------------------------------------------------------------------------------+
# | Starting section test_one |
# +------------------------------------------------------------------------------+
# +..............................................................................+
# : Starting STEP 1: the passed step :
# +..............................................................................+
# The result of STEP 1: the passed step is => PASSED
# +..............................................................................+
# : Starting STEP 2: the failed step :
# +..............................................................................+
# An assertion error was caught during step:
# Traceback (most recent call last):
# File "example.py", line 14, in test_one
# assert 1 == 0
# AssertionError
# The result of STEP 2: the failed step is => FAILED
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - the passed step Passed
# STEP 2 - the failed step Failed
# ------------------------------------------------------------
# The result of section test_one is => FAILED
# +------------------------------------------------------------------------------+
# | Starting section test_two |
# +------------------------------------------------------------------------------+
# +..............................................................................+
# : Starting STEP 1: the errored step :
# +..............................................................................+
# The result of STEP 1: the errored step is => ERRORED
# Caught exception during execution:
# Traceback (most recent call last):
# File "example.py", line 22, in test_two
# {}['non existent key']
# KeyError: 'non existent key'
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - the errored step Errored
# ------------------------------------------------------------
# The result of section test_two is => ERRORED
# The result of testcase Testcase is => ERRORED
# +----------------------------------------------------------------------+
# | Detailed Results |
# +----------------------------------------------------------------------+
# SECTIONS/TESTCASES RESULT
# ------------------------------------------------------------------------
# .
# `-- Testcase ERRORED
# |-- test_one FAILED
# | |-- Step 1: the passed step PASSED
# | |-- Step 2: the failed step PASSED
# `-- test_two ERRORED
# `-- Step 1: the errored step ERRORED
In addition, the Step Objects also offers result APIs to that enables manual assignment of results to each step. Note that these APIs only affect the current step’s result, and do not have Goto support.
Step.passed(reason)
Step.failed(reason)
Step.errored(reason)
Step.skipped(reason)
Step.blocked(reason)
Step.aborted(reason)
Step.passx(reason)
# Example
# -------
#
# step result APIs
from pyats import aetest
class Testcase(aetest.Testcase):
@aetest.test
def test_one(self, steps):
with steps.start('the passed step') as step:
# manually provide Passed result
step.passed('because i want to')
with steps.start('the failed step') as step:
# manually provide Failed result
step.failed('because i had to')
By default, when a step’s result is not Passed, Passx, or Skipped,
all remaining steps are avoided and the engine terminates the current test
section immediately to achieve time-savings. This behavior can be avoided by
providing continue_ = True to steps.start().
# Example
# -------
#
# step continuation feature
from pyats import aetest
class Testcase(aetest.Testcase):
# test section that immediately returns after the first step failure
@aetest.test
def test_stopped_due_to_step_failure(self, steps):
with steps.start('the failed first step'):
# intentionally cause a failure
assert 1 == 0
with steps.start('the step after failed step'):
# do nothing - this would normally be Passed
pass
# same test content/steps, but using continue_ = True
@aetest.test
def test_continues_after_step_failure(self, steps):
with steps.start('the failed first step', continue_ = True):
# intentionally cause a failure
assert 1 == 0
with steps.start('the step after failed step'):
# do nothing - this would normally be Passed
pass
# output of script:
# note that even though both test failed, the 2nd test ran both steps
# because we used continue_ = True to override the default quick-exit
# behavior
#
# +------------------------------------------------------------------------------+
# | Starting section test_stopped_due_to_step_failure |
# +------------------------------------------------------------------------------+
# +..............................................................................+
# : Starting STEP 1: the failed first step :
# +..............................................................................+
# An assertion error was caught during step:
# Traceback (most recent call last):
# File "example.py", line 10, in test_stopped_due_to_step_failure
# assert 1 == 0
# AssertionError
# The result of STEP 1: the failed first step is => FAILED
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - the failed first step Failed
# ------------------------------------------------------------
# The result of section test_stopped_due_to_step_failure is => FAILED
# +------------------------------------------------------------------------------+
# | Starting section test_continues_after_step_failure |
# +------------------------------------------------------------------------------+
# +..............................................................................+
# : Starting STEP 1: the failed first step :
# +..............................................................................+
# An assertion error was caught during step:
# Traceback (most recent call last):
# File "example.py", line 21, in test_continues_after_step_failure
# assert 1 == 0
# AssertionError
# The result of STEP 1: the failed first step is => FAILED
# +..............................................................................+
# : Starting STEP 2: the step after failed step :
# +..............................................................................+
# The result of STEP 2: the step after failed step is => PASSED
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - the failed first step Failed
# STEP 2 - the step after failed step Passed
# ------------------------------------------------------------
# The result of section test_continues_after_step_failure is => FAILED
Note
continue is a python reserved keyword. Following PEP8, the convention
is to add a trailing underscore: continue_.
Nesting Steps¶
Steps can be nested. If a new step is started before the current one finishes,
it is called a child step of the current step. The . separator separates
child indexes from the parent index. Nesting steps provides better visual
references & finer granuarity to the steps breakdown.
Nested step results follow the standard rollup rule: each step’s result is the combined roll-up result of its immediate child steps.
# Example
# -------
#
# step nesting example
# (using a function for demonstration)
from pyats import aetest
# import Steps
from pyats.aetest.steps import Steps
# defining a function that support child steps.
# use Steps() as default value for steps, in case the function is called
# outside the scope of a testscript.
def myFunction(steps = Steps()):
with steps.start('function step one'):
pass
with steps.start('function step two'):
pass
class Testcase(aetest.Testcase):
@aetest.test
def test(self, steps):
# demonstrating a step with multiple child steps
with steps.start('test step 1') as step:
with step.start('test step 1 substep a'):
pass
with step.start('test step 1 substep a') as substep:
with substep.start('test step 1 sub-step a sub-substep i'):
pass
with substep.start('test step 1 sub-step a sub-substep ii'):
pass
# demonstrating a step where a function is called, and
# the function it self takes a few child steps to complete
with steps.start('call function step') as step:
# call the function, pass current step into it
myFunction(step)
# example output:
#
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - test step 1 Passed
# STEP 1.1 - test step 1 substep a Passed
# STEP 1.2 - test step 1 substep a Passed
# STEP 1.2.1 - test step 1 sub-step a sub-substep i Passed
# STEP 1.2.2 - test step 1 sub-step a sub-substep ii Passed
# STEP 2 - call function step Passed
# STEP 2.1 - function step one Passed
# STEP 2.2 - function step two Passed
# ------------------------------------------------------------
Details & Report¶
When steps are created within script sections, a STEPS Report is always logged at the end of that section. This provides visual details in the log file on all steps taken during this section, their names and corresponding results.
This information can also be accessed during runtime using steps attributes:
report()generate the same steps report based on the current step and all of its child steps. This allows users to generate a sliced view the whole picture.
detailsread-only property, returns a list of
StepDetailnamedtuple objects, listing out the current step and all of its child step information. EachStepDetailcontains the following:index- step index stringname- step nameresult- step result
# Example
# -------
#
# accessing step details & localized reports
from pyats import aetest
# import Steps
from pyats.aetest.steps import Steps
class Testcase(aetest.Testcase):
@aetest.test
def test(self, steps):
# creating a step monster
with steps.start('test step 1') as step:
with step.start('test step 1 substep a'):
pass
with step.start('test step 1 substep a') as substep:
with substep.start('test step 1 sub-step a sub-substep i'):
pass
with substep.start('test step 1 sub-step a sub-substep ii'):
pass
# access local step information/report
print(substep.details)
substep.report()
# access overall step details:
print(steps.details)
# output of example
# note the StepDetails printout, and the first STEPS Report (on substep)
#
# +------------------------------------------------------------------------------+
# | Starting section test |
# +------------------------------------------------------------------------------+
# +..............................................................................+
# : Starting STEP 1: test step 1 :
# +..............................................................................+
# +..............................................................................+
# : Starting STEP 1.1: test step 1 substep a :
# +..............................................................................+
# The result of STEP 1.1: test step 1 substep a is => PASSED
# +..............................................................................+
# : Starting STEP 1.2: test step 1 substep a :
# +..............................................................................+
# +..............................................................................+
# : Starting STEP 1.2.1: test step 1 sub-step a sub-substep i :
# +..............................................................................+
# The result of STEP 1.2.1: test step 1 sub-step a sub-substep i is => PASSED
# +..............................................................................+
# : Starting STEP 1.2.2: test step 1 sub-step a sub-substep ii :
# +..............................................................................+
# The result of STEP 1.2.2: test step 1 sub-step a sub-substep ii is => PASSED
# -> [StepDetail(index='1.2', name='test step 1 substep a', result=Passed),
# -> StepDetail(index='1.2.1', name='test step 1 sub-step a sub-substep i', result=Passed),
# -> StepDetail(index='1.2.2', name='test step 1 sub-step a sub-substep ii', result=Passed)]
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1.2 - test step 1 substep a Passed
# STEP 1.2.1 - test step 1 sub-step a sub-substep i Passed
# STEP 1.2.2 - test step 1 sub-step a sub-substep ii Passed
# ------------------------------------------------------------
# The result of STEP 1.2: test step 1 substep a is => PASSED
# The result of STEP 1: test step 1 is => PASSED
# -> [StepDetail(index='1', name='test step 1', result=Passed),
# -> StepDetail(index='1.1', name='test step 1 substep a', result=Passed),
# -> StepDetail(index='1.2', name='test step 1 substep a', result=Passed),
# -> StepDetail(index='1.2.1', name='test step 1 sub-step a sub-substep i', result=Passed),
# -> StepDetail(index='1.2.2', name='test step 1 sub-step a sub-substep ii', result=Passed)]
# +----------------------------------------------------------+
# | STEPS Report |
# +----------------------------------------------------------+
# STEP 1 - test step 1 Passed
# STEP 1.1 - test step 1 substep a Passed
# STEP 1.2 - test step 1 substep a Passed
# STEP 1.2.1 - test step 1 sub-step a sub-substep i Passed
# STEP 1.2.2 - test step 1 sub-step a sub-substep ii Passed
# ------------------------------------------------------------
Step Debugging¶
Step debugging is an optional value-add to steps. It allows the user to send cli commands to currently connected testbed devices and/or run custom debugging functions before and after each step, without modifying the testscript.
To use step debugging, a step debug input file (in YAML format) needs to be
provided to aetest. The content of this file specifies where during script
execution & what clis to send to which testbed devices and which functions to
run.
# Schema
# -------
#
# step debug input file
extends: # Step debug file(s) to extend/build on.
# Use this field to extend an existing yaml step debug file,
# allowing you to create an inheritance hierarchy.
# Supports full path/names or name of file in the same dir.
# The content of the last file on the list forms the base and
# is updated with the preceding file, and so on,
# until the existing file content is updated last.
# (optional)
<step_name>: # A regular expression of the step name
- when: # A list that is consist of 'start', 'end' or any result type,
# like 'passed'
# (mandatory)
device: # List of devices that will be passed to the functions and
# will run the commands.
# (mandatory)
cmd: # List of cli commands to be run on devices,
# (optional)
func: # Functions list,
# (optional)
# Example
# -------
#
# step debug input file
step.*:
- when:
- 'start'
- 'failed'
device:
- 'device.*'
cmd:
- 'mycommand1'
- 'my_command2'
func:
- 'path.to.myscript.myfunction1'
- 'path.to.myscript.myfunction2'
- when:
- 'end'
device:
- 'device1'
cmd:
- 'my_command'
func:
- 'path.to.myscript.myfunction1'
In essence, the step debug engine matches the following items:
the current step name as a regular expression
a particular action:
when encountering a result type using keyword in
when, for example:when('failed')before/after the step using action keyword “start”, “end”
when("start", "end")
and sends a list of cli commands to the given devices. The output of each
command (regardless of error) is then logged to log file for debugging purposes.
Also, users can provide functions/callables to run with the func keyword.
The step debug YAML file is provided to aetest execution using the
Standard Arguments -step_debug. Upon the start of execution,
the input file is loaded & parsed. If there is a missing information in this
file that is mandatory, an exception is thrown. If there are some wrong
information in the file such as wrong device name or step name, it is just
ignored.
# standalone execution example
python my_testscript.py -step_debug /path/to/my/stepDebugInput.yaml
Due to the string matching nature of step debug engine, regular expressions are supported for step names and device names.
About the keys:
whenanddevicekeys are mandatory in the step debug file.cmdandfuncare optional and can be provided together.If
cmdandfuncare provided together, the cli commands are first executed in turn on matching devices and only then are the functions are called on each device in turn.“func” parameter is not limited to just functions, it actually accepts any kind of python callable object(such as classes).
Important
When a function is provided there are 3 keywords that step debug engine is using. If function accepts “section” variable, engine sends the current section object to the function, if “step” is defined then the current step object will also be sent. All of the matching devices are passed to the function one by one, if there is “device” within the function parameters.
If there is a kwargs in the parameters of the function then all of the 3 parameters above will be passed via kwargs.
Step Objects¶
Step feature is internally implemented using two classes:
Stepsbase container class, containing one or more
Step. Allows the creation, reporting and handling of more steps within. Thestepsparameter passed to each section function is an instance of this class.
+--------------------------------------------------------------------------+
| Steps |
+==========================================================================+
| attribute | description |
|-----------------+--------------------------------------------------------|
| start | starts a new step. returns Step instance |
| result | roll-up result of all steps contained |
| report | reports current step details/results to log file |
| steps | list of Step objects representing each step taken |
+==========================================================================+
| properties | description |
|-----------------+--------------------------------------------------------|
| details | list of steps details using StepDetail namedtuple |
+--------------------------------------------------------------------------+
Stepextends the base
Stepsclass.Stepis a Context Manager, intended to be used in conjunction with pythonwithstatement. This is the workhorse class that offers:result apis: manually providing results to each step
error/exception handling: assigns corresponding results to the current step in case of abnormality.
continue on fail (
continue_) featureetc.
Inheriting base class
Stepsenables steps nesting.
+--------------------------------------------------------------------------+
| Step (basecls: Steps) |
+==========================================================================+
| attribute | description |
|-----------------+--------------------------------------------------------|
| start | starts a new child step. returns Step instance |
| result | roll-up result of this step and all child steps ned |
| report | reports current step details/results to log file |
| steps | list of child Step objects |
| description | description of this step instance |
+==========================================================================+
| properties | description |
|-----------------+--------------------------------------------------------|
| details | list of steps details using StepDetail namedtuple |
+==========================================================================+
| result apis | description |
|-----------------+--------------------------------------------------------|
| passed | provides passed result to this step |
| failed | provides failed result to this step |
| aborted | provides aborted result to this step |
| blocked | provides blocked result to this step |
| skipped | provides skipped result to this step |
| errored | provides errored result to this step |
| passx | provides passx result to this step |
+==========================================================================+
| built-in | description |
|-----------------+--------------------------------------------------------|
| __enter__ | method called with starting step through with statement|
| __exit__ | method called with exiting step through with statement |
+--------------------------------------------------------------------------+
Note
the above is for reference only. Do not modify internals during runtime.
Hint
intentionally modifying non-passing results to Passed is cheating. May
be considered a C.L.M.