Automatic Reruns

Rerun Plugin provides an important functionality that allows users to rerun some of their testcases. These testcases are determined either through explicit selection, or the results of the same testcases in a previous run. Any testcases that do not match the provided criteria are not executed and do not appear in the final report.

Rerun Results File

Users may use the --rerun-file argument to provide a “rerun.results” file which details the results of all testcases from a previous run in YAML format. This file is generated by Rerun Plugin when execution is complete. --rerun-file also accepts the “.zip” archive created from a previous run which contains a “rerun.results” file. --rerun-file executes all testcases with a result other than Passed, or as defined by --rerun-condition.

Rerun Condition

--rerun-condition accepts a list of Result Objects to determine which testcases to rerun. The result of each testcase in the “rerun.results” file must match one of the results in the --rerun-condition list to be executed. Passed is not accepted as a rerun condition, and the default list is all result types except Passed.

Rerun Selection File

Users may also wish to specify testcases to rerun regardless of their previous results. To do this, a YAML file with a similar format as “rerun.results” can be created. This file does not hold any testcase results and is not affected by the --rerun-condition list. Testcases are executed based on inclusion in the file. Any testcases to be ignored are simply omitted from the file. This file is specified with --rerun-file just like a “rerun.results” file.

Rerun CLI

Users can alternatively provide a series of CLI arguments to specify tasks and testcases to run. The --rerun-task argument is used once for each task to run, which means it is used multiple times if there are multple tasks to run. The --rerun-task argument is followed by the ID of the task to run, then the path to the testscript of that task, then optionally by any testcases to be run. If no testcases are provided, the entire task will be run. For example, to run tc1, and tc2 from Task-1 and everything from Task-2, the arguments would be:

--rerun-task Task-1 /path/to/script1 tc1 tc2 --rerun-task Task-2 /path/to/script2

File Format

# The task-id is the key for each task
Task-1:
  # the path to this task's testscript is mandatory.
  testscript: /path/to/testscript.py
  # commonSetup and commonCleanup are seperate from Testcases, and they will
  # only be executed if present in the file.
  commonSetup:
    name: common_setup
    result: PASSED
  commonCleanup:
    name: common_cleanup
    result: PASSED
  # list of all testcases in Task-1
  testcases:
  - name: tc_one
    # result is only necessary for a results rerun file. A selection rerun
    # file would omit these.
    result: PASSED
  - name: tc_two
    result: PASSED
Task-2:
  testscript: /path/to/testscript.py
  commonSetup:
    name: common_setup
    result: PASSED
  commonCleanup:
    name: common_setup
    result: PASSED
  testcases:
  - name: tc_one
    result: FAILED
  - name: tc_two
    result: ERRORED

Some notes:

• If Passed is provided as --rerun-condition an exception will be thrown. Because it’s not an allowed result type for Rerun Plugin.

• If neither --rerun-file nor --rerun-task is provided, the entire script will be run again, no matter what the value of --rerun-condition specified.

• The testscript path is manditory for each task passed to Rerun Plugin.

• When rerunning from previous results, if none of the testcases of a task matches the --rerun-condition, that task will be skipped entirely and won’t be included in the reports.

• If the result of common_setup matches the --rerun-condition, all testcases for that task will be executed.

Examples

Rerunning from Results

In this example we will assume that the job file was ran once so that we have the rerun.results. Then, we will run the same job file while using the --rerun-file parameter to pass it.

Job File:

import os
from pyats.easypy import run

# All run() must be inside a main function
def main(runtime):
    # Find the location of the script in relation to the job file
    test_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    testscript = os.path.join(test_path, 'basic_example_script.py')

    # Execute the testscript
    run(testscript=testscript, runtime = runtime)
    run(testscript=testscript, runtime = runtime)

rerun.results:

# rerun.results file that contains the information of each task and their
# testcases.
Task-1:
  commonCleanup: {name: common_cleanup, result: PASSED}
  commonSetup: {name: common_setup, result: PASSED}
  testcases:
  - {name: tc_one, result: FAILED}
  - {name: tc_two, result: PASSED}
  testscript:   path/to/script
Task-2:
  commonCleanup: {name: common_cleanup, result: PASSED}
  commonSetup: {name: common_setup, result: PASSED}
  testcases:
  - {name: tc_one, result: PASSED}
  - {name: tc_two, result: PASSED}
  testscript:   path/to/script

Output:

bash$ pyats run job basic_example_job.py --rerun-file path/to/rerun.results --rerun-condition Failed
%EASYPY-INFO: Starting job run: basic_example_job
%EASYPY-INFO: Starting task execution: Task-1
%EASYPY-INFO:     test harness = ats.aetest
%EASYPY-INFO:     testscript   = path/to/script
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                            Starting common setup                             |
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                   Starting subsection sample_subsection_1                    |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: Aetest Common Setup
%AETEST-INFO: The result of subsection sample_subsection_1 is => PASSED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                   Starting subsection sample_subsection_2                    |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: Inside subsection sample_subsection_2
%SCRIPT-INFO: Inside class common_setup
%AETEST-INFO: The result of subsection sample_subsection_2 is => PASSED
%AETEST-INFO: The result of common setup is => PASSED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                           Starting testcase tc_one                           |
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                      Starting section prepare_testcase                       |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: Preparing the test
%SCRIPT-INFO: section prepare_testcase
%AETEST-INFO: The result of section prepare_testcase is => PASSED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                        Starting section simple_test_1                        |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: First test section
%AETEST-INFO: The result of section simple_test_1 is => PASSED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                        Starting section simple_test_2                        |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: Second test section
%AETEST-INFO: The result of section simple_test_2 is => PASSED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                       Starting section clean_testcase                        |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: Pass testcase cleanup
%AETEST-INFO: The result of section clean_testcase is => PASSED
%AETEST-INFO: The result of testcase tc_one is => PASSED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                           Starting testcase tc_two                           |
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-WARNING: Section 'tc_two' does not match up to: Or('common_setup', 'tc_one', 'common_cleanup')
%AETEST-WARNING: Skipping section...
%AETEST-INFO: The result of testcase tc_two is => SKIPPED
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                           Starting common cleanup                            |
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: +------------------------------------------------------------------------------+
%AETEST-INFO: |                     Starting subsection clean_everything                     |
%AETEST-INFO: +------------------------------------------------------------------------------+
%SCRIPT-INFO: Aetest Common Cleanup
%AETEST-INFO: The result of subsection clean_everything is => PASSED
%AETEST-INFO: The result of common cleanup is => PASSED
%EASYPY-INFO: Task-2 doesn't exist in the rerun_file or it doesn't have any testcase matching the condition: ['FAILED']. Therefore, it will be skipped
%EASYPY-INFO: Job finished. Wrapping up...
%EASYPY-INFO: Creating archive file /path/to/zip/file
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: |                                Easypy Report                                 |
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: pyATS Instance   : /path/to/instance
%EASYPY-INFO: Python Version   : cpython-3.4.1
%EASYPY-INFO: CLI Arguments    : easypy basic_example_job.py
%EASYPY-INFO: User             : tyilder
%EASYPY-INFO: Host Server      : nostg-ott-lnx-1
%EASYPY-INFO: Host OS Version  : Red Hat Enterprise Linux Client 5.5 Tikanga (x86_64)
%EASYPY-INFO:
%EASYPY-INFO: Job Information
%EASYPY-INFO:     Name         : basic_example_job
%EASYPY-INFO:     Start time   : 2016-04-13 17:24:18.439174
%EASYPY-INFO:     Stop time    : 2016-04-13 17:24:18.787723
%EASYPY-INFO:     Elapsed time : 0:00:00.348549
%EASYPY-INFO:     Archive      : /path/to/zip/file
%EASYPY-INFO:
%EASYPY-INFO: Total Tasks    : 1
%EASYPY-INFO:
%EASYPY-INFO: Overall Stats
%EASYPY-INFO:     Passed     : 3
%EASYPY-INFO:     Passx      : 0
%EASYPY-INFO:     Failed     : 0
%EASYPY-INFO:     Aborted    : 0
%EASYPY-INFO:     Blocked    : 0
%EASYPY-INFO:     Skipped    : 1
%EASYPY-INFO:     Errored    : 0
%EASYPY-INFO:
%EASYPY-INFO:     TOTAL      : 4
%EASYPY-INFO:
%EASYPY-INFO: Success Rate   : 100.00 %
%EASYPY-INFO:
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: |                             Task Result Summary                              |
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: Task-1: basic_example_script.commonSetup                                 PASSED
%EASYPY-INFO: Task-1: basic_example_script.tc_one                                      PASSED
%EASYPY-INFO: Task-1: basic_example_script.commonCleanup                               PASSED
%EASYPY-INFO:
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: |                             Task Result Details                              |
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: Task-1: basic_example_script
%EASYPY-INFO: |-- commonSetup                                                           PASSED
%EASYPY-INFO: |   |-- sample_subsection_1                                               PASSED
%EASYPY-INFO: |   `-- sample_subsection_2                                               PASSED
%EASYPY-INFO: |-- tc_one                                                                PASSED
%EASYPY-INFO: |   |-- prepare_testcase                                                  PASSED
%EASYPY-INFO: |   |-- simple_test_1                                                     PASSED
%EASYPY-INFO: |   |-- simple_test_2                                                     PASSED
%EASYPY-INFO: |   `-- clean_testcase                                                    PASSED
%EASYPY-INFO: `-- commonCleanup                                                         PASSED
%EASYPY-INFO:     `-- clean_everything                                                  PASSED

Note

tc_two from Task-1 and the entirety of Task-2 are omitted because they did not match the condition Failed.

Rerun Selection

In this example, assume we have the same job, but we wish to run tc_two for Task-1 and tc_one for Task-2. We could create the following file.

Task-1:
  testcases:
  - name: tc_two
  testscript: path/to/script
Task-2:
  testcases:
  - name: tc_one
  testscript: path/to/script

The final result would look like this:

%EASYPY-INFO: Total Tasks    : 2
%EASYPY-INFO:
%EASYPY-INFO: Overall Stats
%EASYPY-INFO:     Passed     : 6
%EASYPY-INFO:     Passx      : 0
%EASYPY-INFO:     Failed     : 0
%EASYPY-INFO:     Aborted    : 0
%EASYPY-INFO:     Blocked    : 0
%EASYPY-INFO:     Skipped    : 0
%EASYPY-INFO:     Errored    : 0
%EASYPY-INFO:
%EASYPY-INFO:     TOTAL      : 6
%EASYPY-INFO:
%EASYPY-INFO: Success Rate   : 100.00 %
%EASYPY-INFO:
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: |                             Task Result Summary                              |
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: Task-1: basic_example_script.commonSetup                                 PASSED
%EASYPY-INFO: Task-1: basic_example_script.tc_two                                      PASSED
%EASYPY-INFO: Task-1: basic_example_script.commonCleanup                               PASSED
%EASYPY-INFO: Task-2: basic_example_script.commonSetup                                 PASSED
%EASYPY-INFO: Task-2: basic_example_script.tc_one                                      PASSED
%EASYPY-INFO: Task-2: basic_example_script.commonCleanup                               PASSED
%EASYPY-INFO:
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: |                             Task Result Details                              |
%EASYPY-INFO: +------------------------------------------------------------------------------+
%EASYPY-INFO: Task-1: basic_example_script
%EASYPY-INFO: |-- commonSetup                                                           PASSED
%EASYPY-INFO: |   |-- sample_subsection_1                                               PASSED
%EASYPY-INFO: |   `-- sample_subsection_2                                               PASSED
%EASYPY-INFO: |-- tc_two                                                                PASSED
%EASYPY-INFO: |   |-- prepare_testcase                                                  PASSED
%EASYPY-INFO: |   |-- simple_test_1                                                     PASSED
%EASYPY-INFO: |   |-- simple_test_2                                                     PASSED
%EASYPY-INFO: |   `-- clean_testcase                                                    PASSED
%EASYPY-INFO: `-- commonCleanup                                                         PASSED
%EASYPY-INFO:     `-- clean_everything                                                  PASSED
%EASYPY-INFO: Task-2: basic_example_script
%EASYPY-INFO: |-- commonSetup                                                           PASSED
%EASYPY-INFO: |   |-- sample_subsection_1                                               PASSED
%EASYPY-INFO: |   `-- sample_subsection_2                                               PASSED
%EASYPY-INFO: |-- tc_one                                                                PASSED
%EASYPY-INFO: |   |-- prepare_testcase                                                  PASSED
%EASYPY-INFO: |   |-- simple_test_1                                                     PASSED
%EASYPY-INFO: |   |-- simple_test_2                                                     PASSED
%EASYPY-INFO: |   `-- clean_testcase                                                    PASSED
%EASYPY-INFO: `-- commonCleanup                                                         PASSED
%EASYPY-INFO:     `-- clean_everything                                                  PASSED