The Non-Tabular Parser

parsergen is a generic parser for show commands. The goal is to make it simple to create a parser for any given show command one time, and then reuse this parser to create tests for any values found within the output.

The main functions of parsergen are:

• Find and verify expected values in show command output.

• Provide an OS agnostic common interface to OS specific show commands.

• Create once (per command), use many times (throughout all of your pyATS test suites).

• Maintain a per command cache for fast processing.

• Only parse the parts of the output that the user is interested in, thus avoiding wasting precious real-time.

How It Works

parsergen tries to match a given set of data using a set of regular expressions that describe the values found within show command output. The main idea is to exploit the structured form of the output. The regular expression matching is tried through a hierarchical process.

Consider some hypothetical output from a show command:

interface A
    value_1 : 10, value_2 : 20
    value_3 : 30
interface B
    value_1 : 100, value_2 : 200
    value_3 : 300
interface C
    value_1 : 1000, value_2 : 2000
    value_3 : 3000

To access the value_2 of interface A parsergen needs a list of regular expressions to narrow down the scope of the matching. The order of the regexes in the list correspond to the order of the values in the output.

Note that parsergen has a best effort algorithm to catch false positives. For instance, if value_2 is not matched for interface B, parsergen stops the execution as it exits from the inner level.

How To Specify the Regex

Note

Please see extend_markup and mkpgen on how to automatically generate the set of regular expressions for any given show command.

The regular expression must:

• Be identified by a unique key (tag) in the regex dictionary.

• Must have no more than one value extracted per line.

For example:

regex = {
    'iox': {
        'regex1' : "i want to access (this) value in XR"
    },
    'nxos': {
        'regex1' : "i want to access (this) value in NXOS"
    }
}

To teach parsergen a new set of regular expressions use the extend or extend_markup methods.

How To Specify Show Commands

In order to allow for OS agnostic support, and also to allow for migration of show command syntax, one should also teach parsergen about show command syntax. To do this one creates a dictionary that maps a user chosen key value for the show command to the actual show command syntax for a given OS.

For example:

show_cmds = {
    'iox': {
        'SHOW_IP_INTF' : "show {=ipv4} interface {}"
    },
    'nxos': {
        'SHOW_IP_INTF' : "show {=ip} interface {}"
    }
}

The above example now maps the key ‘SHOW_IP_INTF’ to the correct syntax for both XR and NXOS. Additionally the command strings can contain python str.format() arguments. The standard python format function has been extended to allow for specifying default values if the argument is not provided to parsergen. The default values can be seen above for selecting IPv4 interfaces using =ipv4 for XR and =ip for NXOS. If a default value is not provided an empty string is substituted. For requests involving multiple format fields, specify a format argument of None to indicate this is a placeholder for the default field value.

To teach parsergen a new set of show commands pass the dictionary to parsergen.extend along with the set of regular expressions.

parsergen can render CLI commands in an OS-agnostic manner. Here’s an example cli_command_formatting_example.py.

Specifying parsing details using Marked Up Input

The regular expression and show command details mentioned in previous sections can either be specified manually, or can be autogenerated via use of a special CLI command markup format. Please see Output Generated from Marked Up Input for an example of what this autogenerated output looks like.

A Full Example

Warning

Please ensure that all tags are defined for each marked-up value. Do not allow the tool to infer these values from surrounding text. Any rework arising from minor syntax changes in CLI “show” command output from release to release is then isolated to a single parser and will not affect user code.

Warning

Please identify all parts of the router output that could potentially change from run to run and explicitly assign markup tags. Any untagged portions of the markup text is expected to match the router output exactly, otherwise parse failure is expected.

In our full example we will create a parser for the show command show interface. The marked-up output is placed into a file (for example, parsergen_demo_mkpg.py).

from genie import parsergen as pg

marked_up_show_interface_xrvr_output = '''\
OS: iosxr

CMD: show_interface_<WORD>

SHOWCMD: show interface {ifname}

PREFIX: show.intf

ACTUAL:
show interface MgmtEth0/0/CPU0/0
Fri Mar  6 12:03:11.409 EST
MgmtEth0/0/CPU0/0 is up, line protocol is up
  Interface state transitions: 1
  Hardware is Management Ethernet, address is 5254.00d6.36c9 (bia 5254.00d6.36c9)
  Internet address is 10.30.108.132/23
  MTU 1514 bytes, BW 0 Kbit
     reliability 255/255, txload Unknown, rxload Unknown
  Encapsulation ARPA,
  Duplex unknown, 0Kb/s, unknown, link type is autonegotiation
  output flow control is off, input flow control is off
  Carrier delay (up) is 10 msec
  loopback not set,
  ARP type ARPA, ARP timeout 04:00:00
  Last input 00:00:00, output 00:00:48
  Last clearing of "show interface" counters never
  5 minute input rate 79000 bits/sec, 32 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     2459211 packets input, 774707935 bytes, 0 total input drops
     0 drops for unrecognized upper-level protocol
     Received 2216135 broadcast packets, 233738 multicast packets
              0 runts, 0 giants, 0 throttles, 0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     349 packets output, 58930 bytes, 0 total output drops
     Output 4 broadcast packets, 0 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     1 carrier transitions


MARKUP:
show interface MgmtEth0/0/CPU0/0
Fri Mar  6 12:03:11.409 EST
XI<if_name>XMgmtEth0/0/CPU0/0 is XC<admin_state>Xup, line protocol is XW<line_protocol>Xup
  Interface state transitions: XN<intf_trans>X1
  Hardware is XXX<[^,]+><hardware>XXXManagement Ethernet, address is XA<mac_address>X5254.00d6.36c9 (bia 5254.00d6.36c9)
  Internet address is XA<ip_address>X10.30.108.132/23
  MTU XN<mtu>X1514 bytes, BW XNX<bw>0 Kbit
     reliability 255/255, txload Unknown, rxload Unknown
  Encapsulation XW<encap>XARPA,
  Duplex unknown, 0Kb/s, unknown, link type is XW<link_type>Xautonegotiation
  output flow control is XW<output_flowcontrol>Xoff, input flow control is XW<input_flowcontrol>Xoff
  Carrier delay (up) is XN<carrier_delay_up>X10 msec
  loopback XXX<[^,]+><loopback_status>XXXnot set,
  ARP type ARPA, ARP timeout XW<arp_timeout>X04:00:00
  Last input XT<last_input>X00:00:00, output XT<last_output>X00:00:48
  Last clearing of "show interface" counters XR<last_clear_counter>Xnever
  5 minute input rate XN<input_rate_bits>X79000 bits/sec, XN<input_rate>X32 packets/sec
  5 minute output rate XN<output_rate_bits>X0 bits/sec, XN<output_rate>X0 packets/sec
     XN<input_pkts>X2459211 packets input, XN<input_bytes>X774707935 bytes, XN<input_total_drops>X0 total input drops
     XN<drops_unrec_upper_level_proto>X0 drops for unrecognized upper-level protocol
     Received XN<broadcasts>X2216135 broadcast packets, XN<multicasts>X233738 multicast packets
              XN<runts>X0 runts, XN<giants>X0 giants, XN<throttles>X0 throttles, XN<parity>X0 parity
     XN<input_errors>X0 input errors, XN<crc>X0 CRC, XN<frame>X0 frame, XN<overrun>X0 overrun, XN<ignored>X0 ignored, XN<abort>X0 abort
     XN<output_pkts>X349 packets output, XN<output_bytes>X58930 bytes, XN<output_total_drops>X0 total output drops
     Output XN<output_broadcast>X4 broadcast packets, XN<output_multicast>X0 multicast packets
     XN<output_errors>X0 output errors, XN<output_underruns>X0 underruns, XN<output_applique>X0 applique, XN<output_resets>X0 resets
     XN<output_buf_failures>X0 output buffer failures, XN<output_buf_swapped>X0 output buffers swapped out
     XN<carrier_trans>X1 carrier transitions
'''

pg.extend_markup(marked_up_show_interface_xrvr_output)

Registering With Parsergen

Now we extend the current set of regular expressions and show commands recognized by parsergen.

from genie.parsergen.examples.pyAts import parsergen_demo_mkpg

Requesting a Parse and Comparison of Select Tags

In many cases, users are only interested in parsing select portions of the CLI command output.

parsergen offers a comparison mode which allows a list of regex tags to be specified, along with their expected values. A parse is done on these tags only, and if the parsed values are not as expected, parse failure will result.

Here is an example of a parse and compare operation:

device.connect()

attrValPairsToCheck = [
    ('show.intf.if_name',                       'MgmtEth0/0/CPU0/0'),
    ('show.intf.line_protocol',                 'up'),
    ('show.intf.ip_address',                    '10.30.108.132'),
    ('show.intf.mtu',                           1514),
    ('show.intf.admin_state',                   'up'),
]

pgcheck = pg.oper_check (
    device,
    ('show_interface_<WORD>', [], {'ifname':'MgmtEth0/0/CPU0/0'}),
    attrValPairsToCheck,
    refresh_cache=True)

if pgcheck.parse():
    log.info("Parse succeeded.")
    log.info("Parsing details : {}".format(pg.ext_dictio[device.name]))
else:
    log.info("Parse failed.")
    log.error(str(pgcheck))

This API is also predicate-compatible. The above snippet can also be written as such:

from genie.parsergen.predicate import PredicateTestedFalseSignal

attrValPairsToCheck = [
    ('show.intf.if_name',                       'MgmtEth0/0/CPU0/0'),
    ('show.intf.line_protocol',                 'up'),
    ('show.intf.ip_address',                    '10.30.108.132'),
    ('show.intf.mtu',                           1514),
    ('show.intf.admin_state',                   'up'),
]

pgCheckPred = pg.oper_check (
    device,
    ('show_interface_<WORD>', [], {'ifname':'MgmtEth0/0/CPU0/0'}),
    attrValPairsToCheck,
    refresh_cache=True)

try:
    pgCheckPred.assert_test("Parse failed")
    log.info("Parse succeeded.")
    log.info("Parsing details : {}".format(pg.ext_dictio[device.name]))

except PredicateTestedFalseSignal as e:
    log.error(e)

Requesting a Full Parse

Finally we have parsergen generate an object for us that we can ask to parse values from the show output. In this case, we are asking for all the regexp tags that contain the text show.intf to be parsed.

from genie import parsergen as pg

device.connect(alias='myalias')

attrValPairsToParse = [
    ('show.intf.if_name', 'MgmtEth0/0/CPU0/0'),
]
pgfill = pg.oper_fill (
    device=device,
    show_command=\
        ('show_interface_<WORD>', [], {'ifname':'MgmtEth0/0/CPU0/0'}),
    attrvalpairs=attrValPairsToParse,
    refresh_cache=True, regex_tag_fill_pattern='show\.intf',
    device_conn_alias='myalias')
result = pgfill.parse()
if result:
    print("Test passed.  Result :")
    pprint(pg.ext_dictio[device.name]
else:
    print("Parse failed. Diagnosis follows:")
    print(str(pgfill))

Skip Mode

oper_fill has a skip option that can make it possible to parse text that contains optional lines that may or may not be present.

Parsing Output

Here is a sample parsing output obtained by running the command from the previous section:

{'show.intf.abort': '0',
 'show.intf.admin_state': 'up',
 'show.intf.arp_timeout': '04:00:00',
 'show.intf.broadcasts': '16015275',
 'show.intf.bw': '0',
 'show.intf.carrier_delay_up': '10',
 'show.intf.carrier_trans': '1',
 'show.intf.crc': '0',
 'show.intf.drops_unrec_upper_level_proto': '0',
 'show.intf.encap': 'ARPA',
 'show.intf.frame': '0',
 'show.intf.giants': '0',
 'show.intf.hardware': 'Management Ethernet',
 'show.intf.if_name': 'MgmtEth0/0/CPU0/0',
 'show.intf.ignored': '0',
 'show.intf.input_bytes': '5613119704',
 'show.intf.input_errors': '0',
 'show.intf.input_flowcontrol': 'off',
 'show.intf.input_pkts': '17890219',
 'show.intf.input_rate': '33',
 'show.intf.input_rate_bits': '84000',
 'show.intf.input_total_drops': '0',
 'show.intf.intf_trans': '1',
 'show.intf.ip_address': '10.30.108.132',
 'show.intf.last_clear_counter': 'never',
 'show.intf.last_input': '00:00:00',
 'show.intf.last_output': '00:00:48',
 'show.intf.line_protocol': 'up',
 'show.intf.link_type': 'autonegotiation',
 'show.intf.loopback_status': 'not set',
 'show.intf.mac_address': '5254.00d6.36c9',
 'show.intf.mtu': '1514',
 'show.intf.multicasts': '1792005',
 'show.intf.output_applique': '0',
 'show.intf.output_broadcast': '4',
 'show.intf.output_buf_failures': '0',
 'show.intf.output_buf_swapped': '0',
 'show.intf.output_bytes': '1027241',
 'show.intf.output_errors': '0',
 'show.intf.output_flowcontrol': 'off',
 'show.intf.output_multicast': '0',
 'show.intf.output_pkts': '15398',
 'show.intf.output_rate': '0',
 'show.intf.output_rate_bits': '0',
 'show.intf.output_resets': '0',
 'show.intf.output_total_drops': '0',
 'show.intf.output_underruns': '0',
 'show.intf.overrun': '0',
 'show.intf.parity': '0',
 'show.intf.runts': '0',
 'show.intf.throttles': '0'}

Attribute Value Ordering

When using oper_fill or oper_check, ordering of (key,value) pairs is extremely important. Specifying (key,value) pairs in the wrong order can result in parse failure.

If you choose to register using extend_markup, then subsequent (key,value) pairs passed to oper_fill or oper_check will be auto-ordered for you, meaning that you may specify them in any order you prefer.

If you choose to register using extend then you must provide an ordered list of regexp tags if you want the auto-ordering functionality.

Otherwise, if you register using extend but do not provide this ordered list, you must adhere to a strict ordering of (key,value) pairs or face parse failure.

Output Generated from Marked Up Input

Although extend_markup automates the production of all inputs to the extend API, the user still may wish to see this intermediate output for debugging purposes. This is done by running the mkpgen script. For example:

> mkpgen examples/parsergen/pyAts/parsergen_demo_mkpg.py

from genie import parsergen as pgen
show_commands = {
    'iosxr': {
        'show_interface_<WORD>': 'show interface {ifname}',
    },
}
regex = {
    'iosxr': {
        #
        # show_interface_<WORD> ('show interface {ifname}')
        #
        'show.intf.if_name'                       : r'([-A-Za-z0-9\._/:]+) is\s+\w+, line protocol is\s+\w+\s+',
        'show.intf.admin_state'                   : r'[-A-Za-z0-9\._/:]+ is\s+(\w+), line protocol is\s+\w+\s+',
        'show.intf.line_protocol'                 : r'[-A-Za-z0-9\._/:]+ is\s+\w+, line protocol is\s+(\w+)\s+',
        'show.intf.intf_trans'                    : r'\s+Interface state transitions:\s+(\d+)',
        'show.intf.hardware'                      : r'\s+Hardware is\s+([^,]+), address is\s+[A-Fa-f0-9:\.]+ \(bia 5254.00d6.36c9\)',
        'show.intf.mac_address'                   : r'\s+Hardware is\s+[^,]+, address is\s+([A-Fa-f0-9:\.]+) \(bia 5254.00d6.36c9\)',
        'show.intf.ip_address'                    : r'\s+Internet address is\s+([A-Fa-f0-9:\.]+)/23',
        'show.intf.mtu'                           : r'\s+MTU\s+(\d+) bytes, BW\s+\d+ Kbit',
        'show.intf.bw'                            : r'\s+MTU\s+\d+ bytes, BW\s+(\d+) Kbit',
        'show.intf.encap'                         : r'\s+Encapsulation\s+(\w+),',
        'show.intf.link_type'                     : r'\s+Duplex unknown, 0Kb/s, unknown, link type is\s+(\w+)',
        'show.intf.output_flowcontrol'            : r'\s+output flow control is\s+(\w+), input flow control is\s+\w+',
        'show.intf.input_flowcontrol'             : r'\s+output flow control is\s+\w+, input flow control is\s+(\w+)',
        'show.intf.carrier_delay_up'              : r'\s+Carrier delay \(up\) is\s+(\d+) msec',
        'show.intf.loopback_status'               : r'\s+loopback\s+([^,]+),',
        'show.intf.last_clear_counter'            : r'\s+Last clearing of "show interface" counters\s+([^\r\n]+)',
        'show.intf.input_rate_bits'               : r'\s+5 minute input rate\s+(\d+) bits/sec,\s+\d+ packets/sec',
        'show.intf.input_rate'                    : r'\s+5 minute input rate\s+\d+ bits/sec,\s+(\d+) packets/sec',
        'show.intf.output_rate_bits'              : r'\s+5 minute output rate\s+(\d+) bits/sec,\s+\d+ packets/sec',
        'show.intf.output_rate'                   : r'\s+5 minute output rate\s+\d+ bits/sec,\s+(\d+) packets/sec',
        'show.intf.input_pkts'                    : r'\s+(\d+) packets input,\s+\d+ bytes,\s+\d+ total input drops',
        'show.intf.input_bytes'                   : r'\s+\d+ packets input,\s+(\d+) bytes,\s+\d+ total input drops',
        'show.intf.input_total_drops'             : r'\s+\d+ packets input,\s+\d+ bytes,\s+(\d+) total input drops',
        'show.intf.drops_unrec_upper_level_proto' : r'\s+(\d+) drops for unrecognized upper-level protocol',
        'show.intf.broadcasts'                    : r'\s+Received\s+(\d+) broadcast packets,\s+\d+ multicast packets',
        'show.intf.multicasts'                    : r'\s+Received\s+\d+ broadcast packets,\s+(\d+) multicast packets',
        'show.intf.runts'                         : r'\s+(\d+) runts,\s+\d+ giants,\s+\d+ throttles,\s+\d+ parity',
        'show.intf.giants'                        : r'\s+\d+ runts,\s+(\d+) giants,\s+\d+ throttles,\s+\d+ parity',
        'show.intf.throttles'                     : r'\s+\d+ runts,\s+\d+ giants,\s+(\d+) throttles,\s+\d+ parity',
        'show.intf.parity'                        : r'\s+\d+ runts,\s+\d+ giants,\s+\d+ throttles,\s+(\d+) parity',
        'show.intf.input_errors'                  : r'\s+(\d+) input errors,\s+\d+ CRC,\s+\d+ frame,\s+\d+ overrun,\s+\d+ ignored,\s+\d+ abort',
        'show.intf.crc'                           : r'\s+\d+ input errors,\s+(\d+) CRC,\s+\d+ frame,\s+\d+ overrun,\s+\d+ ignored,\s+\d+ abort',
        'show.intf.frame'                         : r'\s+\d+ input errors,\s+\d+ CRC,\s+(\d+) frame,\s+\d+ overrun,\s+\d+ ignored,\s+\d+ abort',
        'show.intf.overrun'                       : r'\s+\d+ input errors,\s+\d+ CRC,\s+\d+ frame,\s+(\d+) overrun,\s+\d+ ignored,\s+\d+ abort',
        'show.intf.ignored'                       : r'\s+\d+ input errors,\s+\d+ CRC,\s+\d+ frame,\s+\d+ overrun,\s+(\d+) ignored,\s+\d+ abort',
        'show.intf.abort'                         : r'\s+\d+ input errors,\s+\d+ CRC,\s+\d+ frame,\s+\d+ overrun,\s+\d+ ignored,\s+(\d+) abort',
        'show.intf.output_pkts'                   : r'\s+(\d+) packets output,\s+\d+ bytes,\s+\d+ total output drops',
        'show.intf.output_bytes'                  : r'\s+\d+ packets output,\s+(\d+) bytes,\s+\d+ total output drops',
        'show.intf.output_total_drops'            : r'\s+\d+ packets output,\s+\d+ bytes,\s+(\d+) total output drops',
        'show.intf.output_broadcast'              : r'\s+Output\s+(\d+) broadcast packets,\s+\d+ multicast packets',
        'show.intf.output_multicast'              : r'\s+Output\s+\d+ broadcast packets,\s+(\d+) multicast packets',
        'show.intf.output_errors'                 : r'\s+(\d+) output errors,\s+\d+ underruns,\s+\d+ applique,\s+\d+ resets',
        'show.intf.output_underruns'              : r'\s+\d+ output errors,\s+(\d+) underruns,\s+\d+ applique,\s+\d+ resets',
        'show.intf.output_applique'               : r'\s+\d+ output errors,\s+\d+ underruns,\s+(\d+) applique,\s+\d+ resets',
        'show.intf.output_resets'                 : r'\s+\d+ output errors,\s+\d+ underruns,\s+\d+ applique,\s+(\d+) resets',
        'show.intf.output_buf_failures'           : r'\s+(\d+) output buffer failures,\s+\d+ output buffers swapped out',
        'show.intf.output_buf_swapped'            : r'\s+\d+ output buffer failures,\s+(\d+) output buffers swapped out',
        'show.intf.carrier_trans'                 : r'\s+(\d+) carrier transitions',

    },
}
regex_tags = {
    'iosxr': [
        #
        # show_interface_<WORD> ('show interface {ifname}')
        #
        'show.intf.if_name'                       ,
        'show.intf.admin_state'                   ,
        'show.intf.line_protocol'                 ,
        'show.intf.intf_trans'                    ,
        'show.intf.hardware'                      ,
        'show.intf.mac_address'                   ,
        'show.intf.ip_address'                    ,
        'show.intf.mtu'                           ,
        'show.intf.bw'                            ,
        'show.intf.encap'                         ,
        'show.intf.link_type'                     ,
        'show.intf.output_flowcontrol'            ,
        'show.intf.input_flowcontrol'             ,
        'show.intf.carrier_delay_up'              ,
        'show.intf.loopback_status'               ,
        'show.intf.last_clear_counter'            ,
        'show.intf.input_rate_bits'               ,
        'show.intf.input_rate'                    ,
        'show.intf.output_rate_bits'              ,
        'show.intf.output_rate'                   ,
        'show.intf.input_pkts'                    ,
        'show.intf.input_bytes'                   ,
        'show.intf.input_total_drops'             ,
        'show.intf.drops_unrec_upper_level_proto' ,
        'show.intf.broadcasts'                    ,
        'show.intf.multicasts'                    ,
        'show.intf.runts'                         ,
        'show.intf.giants'                        ,
        'show.intf.throttles'                     ,
        'show.intf.parity'                        ,
        'show.intf.input_errors'                  ,
        'show.intf.crc'                           ,
        'show.intf.frame'                         ,
        'show.intf.overrun'                       ,
        'show.intf.ignored'                       ,
        'show.intf.abort'                         ,
        'show.intf.output_pkts'                   ,
        'show.intf.output_bytes'                  ,
        'show.intf.output_total_drops'            ,
        'show.intf.output_broadcast'              ,
        'show.intf.output_multicast'              ,
        'show.intf.output_errors'                 ,
        'show.intf.output_underruns'              ,
        'show.intf.output_applique'               ,
        'show.intf.output_resets'                 ,
        'show.intf.output_buf_failures'           ,
        'show.intf.output_buf_swapped'            ,
        'show.intf.carrier_trans'                 ,

    ],
}
pgen.extend (regex, show_commands, regex_tags)