NSO Packages

All user code that needs to run in NSO must be part of a package. A package is basically a directory of files with a fixed file structure, or a tar archive with the same directory layout. A package consists of code, YANG modules, etc., that are needed in order to add an application or function to NSO. Packages are a controlled way to manage loading and versions of custom applications.

Network Element Drivers (NEDs) are also packages. Each NED allows NSO to manage a network device of a specific type. Except for third party YANG NED package which do not contain a YANG device model by default (and must be downloaded and fixed before adding to package), a NED typically contains a device YANG model and the code, specifying how NSO should connect to the device. For NETCONF devices, NSO includes built-in tools to help you build a NED, as described in NED Administration , that you can use, if needed. Otherwise, a third party YANG NED, if available, should be used instead. Vendors, in some cases, provide the required YANG device models but not the entire NED. In practice, all NSO instances use at least one NED. The set of used NED packages depends of the number of different device types the NSO manages.

When NSO starts, it searches for packages to load. The ncs.conf parameter /ncs-config/load-path defines a list of directories. At initial startup, NSO searches these directories for packages, copies the packages to a private directory tree in the directory defined by the /ncs-config/state-dir parameter in ncs.conf, and loads and starts all the packages found. On subsequent startups, NSO will by default only load and start the copied packages. The purpose of this procedure is to make it possible to reliably load new or updated packages while NSO is running, with fallback to the previously existing version of the packages if the reload should fail.

In a "system install" of NSO, packages are always installed (normally by means of symbolic links) in the packages subdirectory of the "run directory", i.e. by default /var/opt/ncs/packages, and the private directory tree is created in the state subdirectory, i.e. by default /var/opt/ncs/state.

Loading of new or updated packages (as well as removal of packages that should no longer be used) can be requested via the reload action - from the NSO CLI:

admin@ncs# packages reload
reload-result {
    package cisco-ios
    result true
}

This request makes NSO copy all packages found in the load path to a temporary version of its private directory, and load the packages from this directory. If the loading is successful, this temporary directory will be made permanent, otherwise the temporary directory is removed and NSO continues to use the previous version of the packages. Thus when updating packages, always update the version in the load path, and request that NSO does the reload via this action.

If the package changes include modified, added, or deleted .fxs files or .ccl files, NSO needs to run a data model upgrade procedure, also called a CDB upgrade. NSO provides a dry-run option to packages reload action to test the upgrade without committing the changes. Using reload dry-run, you can tell if a CDB upgrade is needed or not.

The report all-schema-changes option of the reload action instructs NSO to produce a report of how the current data model schema is being changed. Combined with a dry-run, the report allows you to verify the modifications introduced with the new versions of the packages before actually performing the upgrade.

For a data model upgrade, including a dry-run, all transactions must be closed. In particular, users having CLI sessions in configure mode must exit to operational mode. If there are ongoing commit queue items, and the wait-commit-queue-empty parameter is supplied, it will wait for the items to finish before proceeding the reload. During this time, it will not allow creating any new transactions. Hence, if one of the queue items fails with 'rollback-on-error' option set, the commit queue's rollback will also fail, and the queue item will be locked. In this case, the reload will be canceled. A manual investigation of the failure is needed in order to proceed with the reload.

While the data model upgrade is in progress, all transactions are closed and new transactions are not allowed. This means that starting a new management session, such as a CLI or SSH connection to the NSO, will also fail, producing an error that the node is in upgrade mode.

By default, the reload action will (when needed) wait up to 10 seconds for commit queue to empty (if the wait-commit-queue-empty parameter is entered) and reload to start.

If there are still open transactions at the end of this period, the upgrade will be canceled and the reload operation will fail. The max-wait-time and timeout-action parameters to the action can modify this behaviour. For example, to wait for up to 30 seconds, and forcibly terminate any transactions that still remain open after this period, we can invoke the action as:

admin@ncs# packages reload max-wait-time 30 timeout-action kill

Thus the default values for these parameters are 10 and fail, respectively. In case there are no changes to .fxs or .ccl files, the reload can be carried out without the data model upgrade procedure, and these parameters are ignored, since there is no need to close open transactions.

When reloading packages NSO will give a warning when the upgrade looks "suspicious", i.e. may break some functionality. Note that this is not a strict upgrade validation, but only intended as a hint to NSO administrator early in the upgrade process that something might be wrong. Currently the following scenarios will trigger the warnings:

If a warning has been triggered it is a strong recommendation to fix the root cause. If all of the warnings are intended, it is possible to proceed with "packages reload force" command.

In some specific situations upgrading a package with newly added custom validation points in the data model may produce an error similar to “no registration found for callpoint NEW-VALIDATION/validate” or simply “application communication failure”, resulting in an aborted upgrade. Please see the section called “New Validation Points” in Development Guide on how to proceed.

In some cases we may want NSO to do the same operation as the reload action at NSO startup, i.e. copy all packages from the load path before loading, even though the private directory copy already exists. This can be achieved in the following ways:

Always use one of these methods when upgrading to a new version of NSO in an existing directory structure, to make sure that new packages are loaded together with the other parts of the new system.

If it is known in advance that there were no data model changes, i.e. none of the .fxs or .ccl files changed, and none of the shared JARs changed in a Java package, and the declaration of the components in the package-meta-data.xml is unchanged, then it is possible to do a lightweight package upgrade, called package redeploy. Package redeploy only loads the specified package, unlike packages reload which loads all of the packages found in the load-path.

admin@ncs# packages package mserv redeploy
result true

Redeploying a package allows to reload updated or load new templates, reload private JARs for a Java package or reload the python code which is a part of this package. Only the changed part of the package will be reloaded, e.g. if there were no changes to Python code, but only templates, then the Python VM will not be restarted, but only templates reloaded. The upgrade is not seamless however as the old templates will be unloaded for a short while before the new ones are loaded, so any user of the template during this period of time will fail; same applies to changed Java or Python code. It is hence the responsibility of the user to make sure that the services or other code provided by package is unused while it is being redeployed.

The package redeploy will return true if the package's resulting status after the redeploy is up. Consequently, if the result of the action is false, then it is advised to check the operational status of the package in the package list.

admin@ncs# show packages package mserv oper-status
oper-status file-load-error
oper-status error-info "template3.xml:2 Unknown servicepoint: templ42-servicepoint"

Unlike a full packages reload operation, new NED packages can be loaded into the system without disrupting existing transactions. This is only possible for new packages, since these packages don't yet have any instance data.

The operation is performed through the /packages/add action. No additional input is necessary. The operation scans all the load-paths for any new NED packages and also verifies the existing packages are still present. If packages are modified or deleted, the operation will fail.

Each NED package defines a ned-id, an identifier that is used in selecting the NED for each managed device. A new NED package is therefore a package with a ned-id value that is not already in use.

In addition, the system imposes some additional constraints, so it is not always possible to add just any arbitrary NED. In particular, NED packages can also contain one or more shared data models, such as NED settings or operational data for private use by the NED, that are not specific to each version of a NED package but rather shared between all versions. These are typically placed outside any mountpoint (device-specific data model), extending the NSO schema directly. So, if a NED defines schema nodes outside any mountpoint, there must be no changes to these nodes if they already exist.

Adding a NED package with modified shared data model is therefore not allowed and all shared data models are verified to be identical before a NED package can be added. If they are not, the /packages/add action will fail and you will have to use the /packages/reload command.

admin@ncs# packages add
add-result {
    package router-nc-1.1
    result true
}

The command returns true if the package's resulting status after deployment is up. Likewise, if the result for a package is false, then the package was added but its code has not started successfully and you should check the operational status of the package with the show packages package PKG oper-status command for additional information. You may then use the /packages/package/redeploy action to retry deploying the package's code, once you have corrected the error.

In a "system install" of NSO, management of pre-built packages is supported through a number of actions. This support is not available in a "local install", since it is dependent on the directory structure created by the "system install". Please refer to the YANG submodule $NCS_DIR/src/ncs/yang/tailf-ncs-software.yang for the full details of the functionality described in this section.