Introduction

In software engineering and computer science, Parsing is the mecahnism of translating incomprehensible data to a script readable form. Parsers are the root of automation, without them, automation could not understand the device. There exists multiple ways to parse the device output, with different packages with each their style. There also exist multiple ways to communicate with the device (Cli, Xml, Rest, Yang, etc.) with each providing different structure for the same information!

Metaparser role is to unify those packages, into one location and one structure. A unified collection of parser, which works across multiple parser packages, and across multiple communication protocols and still returns a common structure. Metaparser allows to have one script which works across multiple OS, multiple communication protocol and parsing packages.

Imagine an parsing infrastructure that:

  • Promotes more easily maintainable platform/type/version agnostic testing scripts by deferring operational data parsing to back-end libraries,
  • Harmonizes parsing output among various interface categories, such as CLI, XML and YANG
  • Enforces only enough structure to give the script writer a consistent look and feel across interface categories
  • Is future proof, allowing a multitude of existing and yet-to-be-imagined parsing implementations to coexist in the backend
  • Enables an elastic parsing ecosystem that is simple enough for the novice but feature-rich enough for the power user
  • Leverages the strength of the modern Python-3 language while still allowing bridging/reuse of Cisco’s vast store of legacy TCL-based parsers.
_images/structure.png

Installation

Metaparser is installed with pip install within a sourced pyATS virtual environment.

pip install genie.metaparser

Note

Make sure to source the env.sh(bash)/env.csh(C shell) to setup the pyATS env. For more information about pyATS installation please check the pyATS documentation.

Once installed, it can be imported using import as shown below

# Metaparser
from genie import metaparser

Support

Reach out to contact us for any questions or issues related to the genie.metaparser package.

You can also post questions to the community forum - the support team patrols these forums daily.

Example

_images/Metaparser.png

First, Import Metaparser as illustrated in the below example so it can be used for inheritance for different schema classes. Metaparser schemaengine provides various functionalities for the use within the schema structure as explained in details here schema engine documentation.

In the below example, we are building a parser for three different show commands using different contexts CLI, Yang and XML. The contexts’ implementations are following the same schema/structure so making the parser more rigid to any change which will consequently provide stable/strong test scripts. Different parsing mechanisms (Regex, parsergen, TextFSM, etc.) can be used withing the parser as shown below.

Note

Below example is not a complete one and it was just used for illustration purpose.

Tons of parsers are already built and can be reached in parsers;

Can’t find a parser and want to build one? Visit Contribute in parsers build section for detailed_steps.

Example 1
---------
# Metaparser
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, Any, Optional

# ==============================================================================
# Schema for:
# * 'show bgp vrf <vrf> <address_family>  policy statistics redistribute
# * 'show bgp vrf <vrf> <address_family>  policy statistics dampening'
# * 'show bgp vrf <vrf> <address_family>  policy statistics neighbor <neighbor>'
# ==============================================================================
class ShowBgpPolicyStatisticsSchema(MetaParser):

    schema = {
        'vrf': {
            Any(): {
                Optional('rpm_handle_count'): int,
                Optional('route_map'): {
                    Any():{
                        Any(): {
                            'action': str,
                            'seq_num': int,
                            'total_accept_count': int,
                            'total_reject_count': int,
                            Optional('command'): {
                                'compare_count': int,
                                'match_count': int,
                                'command': str
                            }
                        },
                    },
                }
            },
        }
    }

class ShowBgpPolicyStatistics(ShowBgpPolicyStatisticsSchema):
    """Parser for:
        show bgp [vrf <vrf>] <address_family>  policy statistics redistribute
        show bgp [vrf <vrf>] <address_family>  policy statistics dampening
        show bgp [vrf <vrf>] <address_family>  policy statistics neighbor <neighbor>
        parser class implements detail parsing mechanisms for cli,xml
        and yang output"""

    def cli(self, cmd):

        out = self.device.execute(cmd)

        # Init vars
        ret_dict = {}
        index = 1

        # extract vrf info if specified,
        # if not, vrf is default
        m = re.compile(r'^show +bgp +vrf +(?P<vrf>\S+)').match(cmd)
        if m:
            vrf = m.groupdict()['vrf']
            if vrf == 'all':
                vrf = ''
        else:
            vrf = 'default'

        for line in out.splitlines():
            line = line.strip()

            # Details for VRF default
            p1 = re.compile(r'^Details +for +VRF +'
                             '(?P<vrf>[\w\-]+)$')
            m = p1.match(line)
            if m:
                vrf = m.groupdict()['vrf']
                nei_flag = True
                continue

            # No such neighbor
            if re.compile(r'No +such +neighbor$').match(line):
                nei_flag = False

            # Total count for redistribute rpm handles: 1
            # Total count for neighbor rpm handles: 1
            # Total count for dampening rpm handles: 1
            p2 = re.compile(r'^Total +count +for +(?P<type>\w+) +rpm +handles: +'
                             '(?P<handles>\d+)$')
            m = p2.match(line)

            # BGP policy statistics not available
            p3 = re.compile(r'^BGP +policy +statistics +not +available$')
            m1 = p3.match(line)

            if m or m1:
                if 'vrf' not in ret_dict:
                    ret_dict['vrf'] = {}

                if vrf not in ret_dict['vrf']:
                    ret_dict['vrf'][vrf] = {}

                ret_dict['vrf'][vrf]['rpm_handle_count'] = \
                    int(m.groupdict()['handles']) if m else 0
                continue

            # C: No. of comparisions, M: No. of matches

                ..... Parser continued .......

        return ret_dict


    def xml(self, cmd):
        out = self.device.execute('{} | xml'.format(cmd))

        etree_dict = {}
        neighbor = None
        # Remove junk characters returned by the device
        out = out.replace("]]>]]>", "")
        root = ET.fromstring(out)

        # top table root
        show_root = Common.retrieve_xml_child(root=root, key='show')
        # get xml namespace
        # {http://www.cisco.com/nxos:7.0.3.I7.1.:bgp}
        try:
            m = re.compile(r'(?P<name>\{[\S]+\})').match(show_root.tag)
            namespace = m.groupdict()['name']
        except Exception:
            return etree_dict

        # compare cli command
        Common.compose_compare_command(root=root, namespace=namespace,
                                       expect_command=cmd)

        # get neighbor
        nei = Common.retrieve_xml_child(root=root, key='__XML__PARAM__neighbor-id')

        if hasattr(nei, 'tag'):
            for item in nei.getchildren():
                if '__XML__value' in item.tag:
                    neighbor = item.text
                    continue

                # cover the senario that __readonly__ may be mssing when
                # there are values in the output
                if '__readonly__' in item.tag:
                    root = item.getchildren()[0]
                else:
                    root = item
        else:
            # top table rootl
            root = Common.retrieve_xml_child(root=root, key='TABLE_vrf')

        if not root:
            return etree_dict

        # -----   loop vrf  -----
        for vrf_tree in root.findall('{}ROW_vrf'.format(namespace)):
            # vrf
            try:
                vrf = vrf_tree.find('{}vrf-name-polstats'.format(namespace)).text
            except Exception:
                break

                ..... Parser continued .......

        return etree_dict

    def yang(self):
        """parsing mechanism: yang

        Function yang() defines the yang type output parsing mechanism which
        typically contains 3 steps: executing, transforming, returning
        """

        map_dict = {}
        cmd = '''<native><interface><Vlan/></interface></native>'''
        output = self.device.get(('subtree', cmd))

        for data in output.data:
            for native in data:
                for interface in native:
                    vlan_id = None
                    interface_name = None
                    ip_address = None
                    ip_mask = None
                    for vlan in interface:
                        # Remove the namespace
                        text = vlan.tag[vlan.tag.find('}')+1:]
                        #ydk.models.ned_edison.ned.Native.Interface.Vlan.name
                        #ydk.models.xe_recent_edison.Cisco_IOS_XE_native.Native.Interface.Vlan.name
                        if text == 'name':
                            vlan_id = vlan.text
                            interface_name = 'Vlan' + str(vlan_id)
                            continue

                ..... Parser continued .......

        # Return to caller
        return map_dict

Example 2 (Using Parsergen parsing Mechanism)
---------------------------------------------
# Import parsergen package
import parsergen

# Build Schema
class ShowIpInterfaceBriefSchema(MetaParser):
    """Parser for show ip interface brief"""
    schema = {'interface':
                {Any():
                    {Optional('vlan_id'):
                        {Optional(Any()):
                                {'ip_address': str,
                                 Optional('interface_is_ok'): str,
                                 Optional('method'): str,
                                 Optional('status'): str,
                                 Optional('protocol'): str}
                        },
                     Optional('ip_address'): str,
                     Optional('interface_is_ok'): str,
                     Optional('method'): str,
                     Optional('status'): str,
                     Optional('protocol'): str}
                },
            }

# Build Parser
class ShowIpInterfaceBrief(ShowIpInterfaceBriefSchema):
    """Parser for:
     show ip interface brief
     parser class implements detail parsing mechanisms for cli and yang output.
    """
    #*************************
    # schema - class variable
    #
    # Purpose is to make sure the parser always return the output
    # (nested dict) that has the same data structure across all supported
    # parsing mechanisms (cli(), yang(), xml()).

    def __init__ (self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.cmd = 'show ip interface brief'.format()

    def cli(self):
        """parsing mechanism: cli

        Function cli() defines the cli type output parsing mechanism which
        typically contains 3 steps: exe
        cuting, transforming, returning
        """
        parsed_dict = {}
        output = self.device.execute(self.cmd)

        if output:
            res = parsergen.oper_fill_tabular(device_output=output,
                                              device_os='iosxe',
                                              table_terminal_pattern=r"^\n",
                                              header_fields=
                                               [ "Interface",
                                                 "IP-Address",
                                                 "OK\?",
                                                 "Method",
                                                 "Status",
                                                 "Protocol" ],
                                              label_fields=
                                               [ "Interface",
                                                 "ip_address",
                                                 "interface_is_ok",
                                                 "method",
                                                 "status",
                                                 "protocol" ],
                                              index=[0])

            # Building the schema out o fthe parsergen output
            if res.entries:
                for intf in res.entries:
                    del res.entries[intf]['Interface']

                parsed_dict['interface'] = res.entries
        return (parsed_dict)

    def yang(self):
        """ parsing mechanism: yang

        Function yang() defines the yang type output parsing mechanism which
        typically contains 3 steps: executing, transforming, returning
        """
        pass

    def yang_cli(self):
        cli_output = self.cli()
        yang_output = self.yang()
        merged_output = _merge_dict(yang_output,cli_output)
        return merged_output

Note

For package advanced usage, you can refer to Advanced Usage section.