The Non-Tabular Parser
parsergen is a generic parser for show commands. The goal is to make
it simple to create a parser for any given show command one time, and then
reuse this parser to create tests for any values found within the output.
The main functions of parsergen are:
Find and verify expected values in show command output.
Provide an OS agnostic common interface to OS specific show commands.
Create once (per command), use many times (throughout all of your pyATS test suites).
Maintain a per command cache for fast processing.
Only parse the parts of the output that the user is interested in, thus avoiding wasting precious real-time.
How It Works
parsergen tries to match a given set of data using a set of
regular expressions that describe the values found within show command
output. The main idea is to exploit the structured form of the output. The
regular expression matching is tried through a hierarchical process.
Consider some hypothetical output from a show command:
interface A
value_1 : 10, value_2 : 20
value_3 : 30
interface B
value_1 : 100, value_2 : 200
value_3 : 300
interface C
value_1 : 1000, value_2 : 2000
value_3 : 3000
To access the value_2 of interface A parsergen needs a list
of regular expressions to narrow down the scope of the matching. The
order of the regexes in the list correspond to the order of the values in
the output.
Note that parsergen has a best effort algorithm to catch false
positives. For instance, if value_2 is not matched for interface B,
parsergen stops the execution as it exits from the inner level.
How To Specify the Regex
Note
Please see extend_markup and mkpgen on
how to automatically generate the set of regular expressions for
any given show command.
- The
regular expressionmust: Be identified by a unique key (tag) in the
regexdictionary.Must have no more than one value extracted per line.
For example:
regex = {
'iox': {
'regex1' : "i want to access (this) value in XR"
},
'nxos': {
'regex1' : "i want to access (this) value in NXOS"
}
}
To teach parsergen a new set of regular expressions use the
extend or extend_markup methods.
How To Specify Show Commands
In order to allow for OS agnostic support, and also to allow for migration of
show command syntax, one should also teach parsergen about show
command syntax. To do this one creates a dictionary that maps a user
chosen key value for the show command to the actual show command syntax for a
given OS.
For example:
show_cmds = {
'iox': {
'SHOW_IP_INTF' : "show {=ipv4} interface {}"
},
'nxos': {
'SHOW_IP_INTF' : "show {=ip} interface {}"
}
}
The above example now maps the key ‘SHOW_IP_INTF’ to the correct syntax for
both XR and NXOS. Additionally the command strings can contain python
str.format() arguments. The standard python format function has been
extended to allow for specifying default values if the argument is not provided
to parsergen. The default values can be seen above for selecting IPv4
interfaces using =ipv4 for XR and =ip for NXOS. If a default value is
not provided an empty string is substituted. For requests involving multiple
format fields, specify a format argument of None to indicate this is a
placeholder for the default field value.
To teach parsergen a new set of show commands pass the dictionary to
parsergen.extend
along with the set of regular expressions.
parsergen can render CLI commands in an OS-agnostic manner.
Here’s an example
cli_command_formatting_example.py.
Specifying parsing details using Marked Up Input
The regular expression and show command details mentioned in previous
sections can either be specified manually, or can be autogenerated via use of a
special CLI command markup format. Please see Output Generated from Marked Up Input for an
example of what this autogenerated output looks like.
A Full Example
Warning
Please ensure that all tags are defined for each marked-up value. Do not allow the tool to infer these values from surrounding text. Any rework arising from minor syntax changes in CLI “show” command output from release to release is then isolated to a single parser and will not affect user code.
Warning
Please identify all parts of the router output that could potentially change from run to run and explicitly assign markup tags. Any untagged portions of the markup text is expected to match the router output exactly, otherwise parse failure is expected.
In our full example we will create a parser for the show command show
interface. The marked-up output is placed into
a file (for example,
parsergen_demo_mkpg.py).
from genie import parsergen as pg
marked_up_show_interface_xrvr_output = '''\
OS: iosxr
CMD: show_interface_<WORD>
SHOWCMD: show interface {ifname}
PREFIX: show.intf
ACTUAL:
show interface MgmtEth0/0/CPU0/0
Fri Mar 6 12:03:11.409 EST
MgmtEth0/0/CPU0/0 is up, line protocol is up
Interface state transitions: 1
Hardware is Management Ethernet, address is 5254.00d6.36c9 (bia 5254.00d6.36c9)
Internet address is 10.30.108.132/23
MTU 1514 bytes, BW 0 Kbit
reliability 255/255, txload Unknown, rxload Unknown
Encapsulation ARPA,
Duplex unknown, 0Kb/s, unknown, link type is autonegotiation
output flow control is off, input flow control is off
Carrier delay (up) is 10 msec
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:48
Last clearing of "show interface" counters never
5 minute input rate 79000 bits/sec, 32 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
2459211 packets input, 774707935 bytes, 0 total input drops
0 drops for unrecognized upper-level protocol
Received 2216135 broadcast packets, 233738 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
349 packets output, 58930 bytes, 0 total output drops
Output 4 broadcast packets, 0 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
MARKUP:
show interface MgmtEth0/0/CPU0/0
Fri Mar 6 12:03:11.409 EST
XI<if_name>XMgmtEth0/0/CPU0/0 is XC<admin_state>Xup, line protocol is XW<line_protocol>Xup
Interface state transitions: XN<intf_trans>X1
Hardware is XXX<[^,]+><hardware>XXXManagement Ethernet, address is XA<mac_address>X5254.00d6.36c9 (bia 5254.00d6.36c9)
Internet address is XA<ip_address>X10.30.108.132/23
MTU XN<mtu>X1514 bytes, BW XNX<bw>0 Kbit
reliability 255/255, txload Unknown, rxload Unknown
Encapsulation XW<encap>XARPA,
Duplex unknown, 0Kb/s, unknown, link type is XW<link_type>Xautonegotiation
output flow control is XW<output_flowcontrol>Xoff, input flow control is XW<input_flowcontrol>Xoff
Carrier delay (up) is XN<carrier_delay_up>X10 msec
loopback XXX<[^,]+><loopback_status>XXXnot set,
ARP type ARPA, ARP timeout XW<arp_timeout>X04:00:00
Last input XT<last_input>X00:00:00, output XT<last_output>X00:00:48
Last clearing of "show interface" counters XR<last_clear_counter>Xnever
5 minute input rate XN<input_rate_bits>X79000 bits/sec, XN<input_rate>X32 packets/sec
5 minute output rate XN<output_rate_bits>X0 bits/sec, XN<output_rate>X0 packets/sec
XN<input_pkts>X2459211 packets input, XN<input_bytes>X774707935 bytes, XN<input_total_drops>X0 total input drops
XN<drops_unrec_upper_level_proto>X0 drops for unrecognized upper-level protocol
Received XN<broadcasts>X2216135 broadcast packets, XN<multicasts>X233738 multicast packets
XN<runts>X0 runts, XN<giants>X0 giants, XN<throttles>X0 throttles, XN<parity>X0 parity
XN<input_errors>X0 input errors, XN<crc>X0 CRC, XN<frame>X0 frame, XN<overrun>X0 overrun, XN<ignored>X0 ignored, XN<abort>X0 abort
XN<output_pkts>X349 packets output, XN<output_bytes>X58930 bytes, XN<output_total_drops>X0 total output drops
Output XN<output_broadcast>X4 broadcast packets, XN<output_multicast>X0 multicast packets
XN<output_errors>X0 output errors, XN<output_underruns>X0 underruns, XN<output_applique>X0 applique, XN<output_resets>X0 resets
XN<output_buf_failures>X0 output buffer failures, XN<output_buf_swapped>X0 output buffers swapped out
XN<carrier_trans>X1 carrier transitions
'''
pg.extend_markup(marked_up_show_interface_xrvr_output)
Registering With Parsergen
Now we extend the current set of regular expressions
and show commands recognized by parsergen.
from genie.parsergen.examples.pyAts import parsergen_demo_mkpg
Requesting a Full Parse
Finally we have parsergen generate an object for us that we can ask to
parse values from the show output. In this case, we are asking for all
the regexp tags that contain the text show.intf to be parsed.
from genie import parsergen as pg
device.connect(alias='myalias')
attrValPairsToParse = [
('show.intf.if_name', 'MgmtEth0/0/CPU0/0'),
]
pgfill = pg.oper_fill (
device=device,
show_command=\
('show_interface_<WORD>', [], {'ifname':'MgmtEth0/0/CPU0/0'}),
attrvalpairs=attrValPairsToParse,
refresh_cache=True, regex_tag_fill_pattern='show\.intf',
device_conn_alias='myalias')
result = pgfill.parse()
if result:
print("Test passed. Result :")
pprint(pg.ext_dictio[device.name]
else:
print("Parse failed. Diagnosis follows:")
print(str(pgfill))
Skip Mode
oper_fill has a skip option that can make it possible to parse text
that contains optional lines that may or may not be present.
Parsing Output
Here is a sample parsing output obtained by running the command from the previous section:
{'show.intf.abort': '0',
'show.intf.admin_state': 'up',
'show.intf.arp_timeout': '04:00:00',
'show.intf.broadcasts': '16015275',
'show.intf.bw': '0',
'show.intf.carrier_delay_up': '10',
'show.intf.carrier_trans': '1',
'show.intf.crc': '0',
'show.intf.drops_unrec_upper_level_proto': '0',
'show.intf.encap': 'ARPA',
'show.intf.frame': '0',
'show.intf.giants': '0',
'show.intf.hardware': 'Management Ethernet',
'show.intf.if_name': 'MgmtEth0/0/CPU0/0',
'show.intf.ignored': '0',
'show.intf.input_bytes': '5613119704',
'show.intf.input_errors': '0',
'show.intf.input_flowcontrol': 'off',
'show.intf.input_pkts': '17890219',
'show.intf.input_rate': '33',
'show.intf.input_rate_bits': '84000',
'show.intf.input_total_drops': '0',
'show.intf.intf_trans': '1',
'show.intf.ip_address': '10.30.108.132',
'show.intf.last_clear_counter': 'never',
'show.intf.last_input': '00:00:00',
'show.intf.last_output': '00:00:48',
'show.intf.line_protocol': 'up',
'show.intf.link_type': 'autonegotiation',
'show.intf.loopback_status': 'not set',
'show.intf.mac_address': '5254.00d6.36c9',
'show.intf.mtu': '1514',
'show.intf.multicasts': '1792005',
'show.intf.output_applique': '0',
'show.intf.output_broadcast': '4',
'show.intf.output_buf_failures': '0',
'show.intf.output_buf_swapped': '0',
'show.intf.output_bytes': '1027241',
'show.intf.output_errors': '0',
'show.intf.output_flowcontrol': 'off',
'show.intf.output_multicast': '0',
'show.intf.output_pkts': '15398',
'show.intf.output_rate': '0',
'show.intf.output_rate_bits': '0',
'show.intf.output_resets': '0',
'show.intf.output_total_drops': '0',
'show.intf.output_underruns': '0',
'show.intf.overrun': '0',
'show.intf.parity': '0',
'show.intf.runts': '0',
'show.intf.throttles': '0'}
Attribute Value Ordering
When using oper_fill or oper_check, ordering of (key,value) pairs is
extremely important.
Specifying (key,value) pairs in the wrong order can result in parse failure.
If you choose to register using extend_markup, then subsequent (key,value)
pairs passed to oper_fill or oper_check will be auto-ordered for you,
meaning that you may specify them in any order you prefer.
If you choose to register using extend then you must provide an ordered
list of regexp tags if you want the auto-ordering functionality.
Otherwise, if you register using extend but do not provide this ordered
list, you must adhere to a strict ordering of (key,value) pairs or face
parse failure.
Output Generated from Marked Up Input
Although extend_markup automates the production of all inputs to the
extend API, the user still may wish to see this intermediate output for
debugging purposes. This is done by running the
mkpgen script. For example:
> mkpgen examples/parsergen/pyAts/parsergen_demo_mkpg.py
from genie import parsergen as pgen
show_commands = {
'iosxr': {
'show_interface_<WORD>': 'show interface {ifname}',
},
}
regex = {
'iosxr': {
#
# show_interface_<WORD> ('show interface {ifname}')
#
'show.intf.if_name' : r'([-A-Za-z0-9\._/:]+) is\s+\w+, line protocol is\s+\w+\s+',
'show.intf.admin_state' : r'[-A-Za-z0-9\._/:]+ is\s+(\w+), line protocol is\s+\w+\s+',
'show.intf.line_protocol' : r'[-A-Za-z0-9\._/:]+ is\s+\w+, line protocol is\s+(\w+)\s+',
'show.intf.intf_trans' : r'\s+Interface state transitions:\s+(\d+)',
'show.intf.hardware' : r'\s+Hardware is\s+([^,]+), address is\s+[A-Fa-f0-9:\.]+ \(bia 5254.00d6.36c9\)',
'show.intf.mac_address' : r'\s+Hardware is\s+[^,]+, address is\s+([A-Fa-f0-9:\.]+) \(bia 5254.00d6.36c9\)',
'show.intf.ip_address' : r'\s+Internet address is\s+([A-Fa-f0-9:\.]+)/23',
'show.intf.mtu' : r'\s+MTU\s+(\d+) bytes, BW\s+\d+ Kbit',
'show.intf.bw' : r'\s+MTU\s+\d+ bytes, BW\s+(\d+) Kbit',
'show.intf.encap' : r'\s+Encapsulation\s+(\w+),',
'show.intf.link_type' : r'\s+Duplex unknown, 0Kb/s, unknown, link type is\s+(\w+)',
'show.intf.output_flowcontrol' : r'\s+output flow control is\s+(\w+), input flow control is\s+\w+',
'show.intf.input_flowcontrol' : r'\s+output flow control is\s+\w+, input flow control is\s+(\w+)',
'show.intf.carrier_delay_up' : r'\s+Carrier delay \(up\) is\s+(\d+) msec',
'show.intf.loopback_status' : r'\s+loopback\s+([^,]+),',
'show.intf.last_clear_counter' : r'\s+Last clearing of "show interface" counters\s+([^\r\n]+)',
'show.intf.input_rate_bits' : r'\s+5 minute input rate\s+(\d+) bits/sec,\s+\d+ packets/sec',
'show.intf.input_rate' : r'\s+5 minute input rate\s+\d+ bits/sec,\s+(\d+) packets/sec',
'show.intf.output_rate_bits' : r'\s+5 minute output rate\s+(\d+) bits/sec,\s+\d+ packets/sec',
'show.intf.output_rate' : r'\s+5 minute output rate\s+\d+ bits/sec,\s+(\d+) packets/sec',
'show.intf.input_pkts' : r'\s+(\d+) packets input,\s+\d+ bytes,\s+\d+ total input drops',
'show.intf.input_bytes' : r'\s+\d+ packets input,\s+(\d+) bytes,\s+\d+ total input drops',
'show.intf.input_total_drops' : r'\s+\d+ packets input,\s+\d+ bytes,\s+(\d+) total input drops',
'show.intf.drops_unrec_upper_level_proto' : r'\s+(\d+) drops for unrecognized upper-level protocol',
'show.intf.broadcasts' : r'\s+Received\s+(\d+) broadcast packets,\s+\d+ multicast packets',
'show.intf.multicasts' : r'\s+Received\s+\d+ broadcast packets,\s+(\d+) multicast packets',
'show.intf.runts' : r'\s+(\d+) runts,\s+\d+ giants,\s+\d+ throttles,\s+\d+ parity',
'show.intf.giants' : r'\s+\d+ runts,\s+(\d+) giants,\s+\d+ throttles,\s+\d+ parity',
'show.intf.throttles' : r'\s+\d+ runts,\s+\d+ giants,\s+(\d+) throttles,\s+\d+ parity',
'show.intf.parity' : r'\s+\d+ runts,\s+\d+ giants,\s+\d+ throttles,\s+(\d+) parity',
'show.intf.input_errors' : r'\s+(\d+) input errors,\s+\d+ CRC,\s+\d+ frame,\s+\d+ overrun,\s+\d+ ignored,\s+\d+ abort',
'show.intf.crc' : r'\s+\d+ input errors,\s+(\d+) CRC,\s+\d+ frame,\s+\d+ overrun,\s+\d+ ignored,\s+\d+ abort',
'show.intf.frame' : r'\s+\d+ input errors,\s+\d+ CRC,\s+(\d+) frame,\s+\d+ overrun,\s+\d+ ignored,\s+\d+ abort',
'show.intf.overrun' : r'\s+\d+ input errors,\s+\d+ CRC,\s+\d+ frame,\s+(\d+) overrun,\s+\d+ ignored,\s+\d+ abort',
'show.intf.ignored' : r'\s+\d+ input errors,\s+\d+ CRC,\s+\d+ frame,\s+\d+ overrun,\s+(\d+) ignored,\s+\d+ abort',
'show.intf.abort' : r'\s+\d+ input errors,\s+\d+ CRC,\s+\d+ frame,\s+\d+ overrun,\s+\d+ ignored,\s+(\d+) abort',
'show.intf.output_pkts' : r'\s+(\d+) packets output,\s+\d+ bytes,\s+\d+ total output drops',
'show.intf.output_bytes' : r'\s+\d+ packets output,\s+(\d+) bytes,\s+\d+ total output drops',
'show.intf.output_total_drops' : r'\s+\d+ packets output,\s+\d+ bytes,\s+(\d+) total output drops',
'show.intf.output_broadcast' : r'\s+Output\s+(\d+) broadcast packets,\s+\d+ multicast packets',
'show.intf.output_multicast' : r'\s+Output\s+\d+ broadcast packets,\s+(\d+) multicast packets',
'show.intf.output_errors' : r'\s+(\d+) output errors,\s+\d+ underruns,\s+\d+ applique,\s+\d+ resets',
'show.intf.output_underruns' : r'\s+\d+ output errors,\s+(\d+) underruns,\s+\d+ applique,\s+\d+ resets',
'show.intf.output_applique' : r'\s+\d+ output errors,\s+\d+ underruns,\s+(\d+) applique,\s+\d+ resets',
'show.intf.output_resets' : r'\s+\d+ output errors,\s+\d+ underruns,\s+\d+ applique,\s+(\d+) resets',
'show.intf.output_buf_failures' : r'\s+(\d+) output buffer failures,\s+\d+ output buffers swapped out',
'show.intf.output_buf_swapped' : r'\s+\d+ output buffer failures,\s+(\d+) output buffers swapped out',
'show.intf.carrier_trans' : r'\s+(\d+) carrier transitions',
},
}
regex_tags = {
'iosxr': [
#
# show_interface_<WORD> ('show interface {ifname}')
#
'show.intf.if_name' ,
'show.intf.admin_state' ,
'show.intf.line_protocol' ,
'show.intf.intf_trans' ,
'show.intf.hardware' ,
'show.intf.mac_address' ,
'show.intf.ip_address' ,
'show.intf.mtu' ,
'show.intf.bw' ,
'show.intf.encap' ,
'show.intf.link_type' ,
'show.intf.output_flowcontrol' ,
'show.intf.input_flowcontrol' ,
'show.intf.carrier_delay_up' ,
'show.intf.loopback_status' ,
'show.intf.last_clear_counter' ,
'show.intf.input_rate_bits' ,
'show.intf.input_rate' ,
'show.intf.output_rate_bits' ,
'show.intf.output_rate' ,
'show.intf.input_pkts' ,
'show.intf.input_bytes' ,
'show.intf.input_total_drops' ,
'show.intf.drops_unrec_upper_level_proto' ,
'show.intf.broadcasts' ,
'show.intf.multicasts' ,
'show.intf.runts' ,
'show.intf.giants' ,
'show.intf.throttles' ,
'show.intf.parity' ,
'show.intf.input_errors' ,
'show.intf.crc' ,
'show.intf.frame' ,
'show.intf.overrun' ,
'show.intf.ignored' ,
'show.intf.abort' ,
'show.intf.output_pkts' ,
'show.intf.output_bytes' ,
'show.intf.output_total_drops' ,
'show.intf.output_broadcast' ,
'show.intf.output_multicast' ,
'show.intf.output_errors' ,
'show.intf.output_underruns' ,
'show.intf.output_applique' ,
'show.intf.output_resets' ,
'show.intf.output_buf_failures' ,
'show.intf.output_buf_swapped' ,
'show.intf.carrier_trans' ,
],
}
pgen.extend (regex, show_commands, regex_tags)