Section Results¶
The outcome of all testing is some form of results. This section thus explores
topics such as how results are propagated & stored internally in aetest, how
exceptions are handled, and how users may override the default result reporting
behavior.
Internally in aetest, all results are collected, rolled-up and reported
using Result Objects. The following summarizes this behavior:
all script section have a
resultattribute, storing its current result using the corresponding result object (see Object Model).TestContainersection’sresultattribute represents the combined roll-up of all of its child section results. Eg, common setup’sresultattribute stores the current combined rolled-up result of all of its subsections that ran so far.
Result Behavior¶
Ei incumbit probatio qui dicit, non qui negat. (Latin)
The default result for all sections is Passed, even if no meaningful actions
and/or testings were carried out. Think of this as: even though a test that
assesses nothing is a terrible test, running it still yields a “passing” grade.
Do not mix-up the usefulness/meaningfulness of a test with the result of its
testing. This behavior is inline with all other Python test infrastructures.
# Example
# -------
#
# default section result is always Passed
from pyats import aetest
# this testcase is entirely empty
class Testcase(aetest.Testcase):
pass
# but if we instantiate this testcase and run it
# we get Passed, even though it did nothing.
tc = Testcase()
tc()
# Passed
When python Exceptions are raised during the execution of any test sections and
are caught by the aetest infrastructure, depending on the type of exception,
a corresponding result is assigned to that running section:
AssertionError: AssertionError exceptions and all of its subclasses corresponds to section resultFailed, indicating a failed assertion test.Example: the test code is using
assertstatement to perform a check/test, and the exception is raised as a result of the failed assertion.
Exception: Exception and all of its subclasses corresponds to section resultErrored, indicating of occurance of an un-handled test-code error.Example: a
KeyErrorexception is raised when the script is accessing a dictionary key that does not exist.
# Example
# -------
#
# demonstrating exception result behaviors
from pyats import aetest
class Testcase(aetest.Testcase):
# defining a test that raises a python exception
# the expected behavior is test Errored
@aetest.test
def test_one(self):
# creating an empty dictionary and accessing a key
# that does not exist raises KeyError Exception
{}['key does not exist']
# defining a test that raises an AssertionError
# the expected behavior is test Failed
@aetest.test
def test_two(self):
# do an assertion that fails.
assert 1 == 0, "unfortunately 1 doesn't equal to 0"
# output of this script
# ---------------------
#
# +------------------------------------------------------------------------------+
# | Starting testcase Testcase |
# +------------------------------------------------------------------------------+
# +------------------------------------------------------------------------------+
# | Starting section test_one |
# +------------------------------------------------------------------------------+
# Caught exception: 'key does not exist'
# Traceback (most recent call last):
# File "examples.py", line 11, in test_one
# {}['key does not exist']
# KeyError: 'key does not exist'
# The result of section test_one is => ERRORED
# +------------------------------------------------------------------------------+
# | Starting section test_two |
# +------------------------------------------------------------------------------+
# An assertion exception was caught.
# unfortunately 1 doesn't equal to 0
# The result of section test_two is => FAILED
# The result of testcase Testcase is => ERRORED
#
# Note: the overall result of this testcase is Errored due to roll-ups
#
# SECTIONS/TESTCASES RESULT
# --------------------------------------------------------------------------
# .
# `-- Testcase ERRORED
# |-- test_one ERRORED
# `-- test_two FAILED
Within your test sections, use try ... except ... statements to handle any
exceptions that are expected. This makes sure these expected exceptions
do not propagate to the test infrastructure, polluting the results of current
running sections.
Note
Beware of the Exception Hierarchy. Do not catch BaseException, as it
causes your scripts to ignore requests to exit and keyboard interrupts
such as ctrl-c.
Result APIs¶
In addition to the automatic result assignment behaviors, it is also possible to
manually provide section results by calling one of the TestItem static
methods:
TestItem.passed(reason, goto, from_exception, data)
TestItem.failed(reason, goto, from_exception, data)
TestItem.errored(reason, goto, from_exception, data)
TestItem.skipped(reason, goto, from_exception, data)
TestItem.blocked(reason, goto, from_exception, data)
TestItem.aborted(reason, goto, from_exception, data)
TestItem.passx(reason, goto, from_exception, data)
Upon calling, the current section execution terminates immediately,
returns and is set with the corresponding result. In other words, result apis
can only be called once per script section, and all code immediately after
it is not executed (similar to how return statement works).
All results apis accept the following optional arguments:
reason, describing the conditions & reasons of why this result is provided.
goto, list of sections to “go to” after this section. Refer to Goto documentation for details.
from_exception, accepts an exception object and will add the traceback of this exception to the result’sreason.
data, accepts a dict of arbitrary data that is relevant to the result. A representation of this data is stored by the Reporter for external processing or reference.
Tip
TestItem is the base class of all classes, and thus you can call
self.failed() within section code directly instead of the absolute
reference TestItem.failed() (see Object Model).
# Example
# -------
#
# manually setting results for sections
from pyats import aetest
# using common setup as an example for a change
class CommonSetup(aetest.CommonSetup):
@aetest.subsection
def subsection_that_passes(self):
self.passed("failure is not an option")
# any code after a result api is not executed
evenThisAPICallThatDoesNotExist()
@aetest.subsection
def subsection_that_fails(self):
self.failed("failure... is a must in this test")
@aetest.subsection
def subsection_that_is_skipped(self):
self.skipped("i don't want to run this section.")
# output of this script
# ---------------------
#
# +------------------------------------------------------------------------------+
# | Starting common setup |
# +------------------------------------------------------------------------------+
# +------------------------------------------------------------------------------+
# | Starting subsection subsection_that_passes |
# +------------------------------------------------------------------------------+
# Passed reason: failure is not an option
# The result of subsection subsection_that_passes is => PASSED
# +------------------------------------------------------------------------------+
# | Starting subsection subsection_that_fails |
# +------------------------------------------------------------------------------+
# Failed reason: failure... is a must in this test
# The result of subsection subsection_that_fails is => FAILED
# +------------------------------------------------------------------------------+
# | Starting subsection subsection_that_is_skipped |
# +------------------------------------------------------------------------------+
# Skipped reason: i don't want to run this section.
# The result of subsection subsection_that_is_skipped is => SKIPPED
# The result of common setup is => FAILED
#
# SECTIONS/TESTCASES RESULT
# -------------------------------------------------------------------------
# .
# `-- CommonSetup FAILED
# |-- subsection_that_passes PASSED
# |-- subsection_that_fails FAILED
# `-- subsection_that_is_skipped SKIPPED
Interaction Results¶
There are also rare cases when a result must be manually determined by a human. Such as a test that relies upon changes that happen in the physical world. For these cases there is a utility that allows a user to decide the result.
The class WebInteraction can pause test execution and notify a user via
email that input is required. This email has a link to a webpage hosted by
WebInteraction that has a form for the user to submit to give a result.
# Example
# -------
#
# example testscript for using WebInteraction
from pyats import aetest
from pyats.aetest.utils.interaction import WebInteraction
class TestcaseOne(aetest.Testcase):
@aetest.setup
def setup(self): pass
@aetest.test
def test_one(self, section):
WebInteraction('Brief title for this interaction',
'Message for the user about how to assess this test',
section = section,
timeout = 300
).interact()
@aetest.cleanup
def cleanup(self): pass
Argument |
Type |
Description |
|---|---|---|
|
|
A brief description of the interaction. |
|
|
This is the message that describes to the user what the test is and how they determine the result. |
|
|
Reference to section or step that called interaction. This provides the name name of the test to the user, as well as the APIs to return a test result. |
|
|
The host address to bind to. |
|
|
The port for the webpage to be hosted at. Default is 0 for a kernel provisioned port. |
|
|
Template for the HTML webpage. This can inherit by extending default_web_template. |
|
|
How long the test should wait for a human in seconds. Can also take string ‘inf’ for no timeout. Default is 1 hour. |
|
|
Name of the result given to the test when
it times out. Default is |
|
|
When set to |
|
|
The email address that the notification will be sent from. Defaults to the user login. |
|
|
The email address that the notification will be sent to. Defaults to the user login. |
|
|
Alternate Subject line for email notification. |
|
|
Alternate Body for email notification. |
Note
web_template can inherit from the existing default template. You can
find this template at
<python_install>/site-packages/pyats/aetest/utils/templates/default_web_template.html
to see what blocks are defined, as well as some arguments that can be
used.
Read about jinja2 template inheritance.
Result Counting¶
In aetest, only TestContainer class’ results counts in the summary
result numbers. Even though child sections within TestContainer classes
have their own results, they are considered to be a part of its parent
container, and their results is thus not counted for in the summary.
- Accounted For
CommonSetup,TestcaseandCommonCleanup.- Not Accounted For
Subsection,SetupSection,TestSectionandCleanupSections.
Condition |
Result Numbers |
|---|---|
|
1 |
|
1 |
2x |
3 |
|
22 |
# Example
# -------
#
# example testscript and number counting
from pyats import aetest
class CommonSetup(aetest.CommonSetup):
@aetest.subsection
def subsection_one(self): pass
@aetest.subsection
def subsection_two(self): pass
class TestcaseOne(aetest.Testcase):
@aetest.setup
def setup(self): pass
@aetest.test
def test_one(self): pass
@aetest.test
def test_two(self):
self.failed()
@aetest.cleanup
def cleanup(self): pass
# +------------------------------------------------------------------------+
# | Summary |
# +------------------------------------------------------------------------+
# Number of ABORTED 0
# Number of BLOCKED 0
# Number of ERRORED 0
# Number of FAILED 1
# Number of PASSED 1
# Number of PASSX 0
# Number of SKIPPED 0
# --------------------------------------------------------------------------
Result API Internals¶
Result APIs such as TestItem.passed() terminate current execution and
return immediately because they are internally implemented to raise
AEtestInternalSignals exceptions, and are handled by the execution engine to
assign a corresponding result to the current test section.
AEtestInternalSignals subclasses from BaseException class intentionally
in order to avoid any blanket catching of Exception in try ... except
clause. They are called signals because they are used for signalling
purposes, to instruct the infrastructure to assign a result and carry on.
Result API |
Exception Signal |
Result Object |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In essence, within anywhere in the testscript and/or libraries, if these
exceptions are raised (as they are still exceptions in nature), aetest
behaves exactly the same as calling result apis.
# Example
# -------
#
# raising signalling exceptions
from pyats import aetest
from pyats.aetest.signals import AEtestAbortedSignal
class CommonCleanup(aetest.CommonCleanup):
@aetest.subsection
def subsection(self):
# subsection getting aborted using signaling
raise AEtestAbortedSignal(reason = 'feeling it.')
# output of this script
# ---------------------
#
# +------------------------------------------------------------------------------+
# | Starting common cleanup |
# +------------------------------------------------------------------------------+
# +------------------------------------------------------------------------------+
# | Starting subsection subsection |
# +------------------------------------------------------------------------------+
# Aborted reason: feeling it.
# The result of subsection subsection is => ABORTED
# The result of common cleanup is => ABORTED