Deployment Example

An admin user is created on the NSO nodes.

Password-less sudo access is set up to enable the tailf-hcc server to run the ip command.

The manager's SSH client uses public key authentication, while the RESTCONF client uses a token to authenticate with the NSO nodes.

The example creates two packages using the ncs-make-package command: dummy and inert. A third package, tailf-hcc, provides VIPs that point to the current HA leader/primary node.

The packages are compressed into a tar.gz format for easier distribution, but that is not a requirement.

NSO's basic dependency requirements are fulfilled by adding the Java Runtime Environment (JRE), OpenSSH, and OpenSSL packages.

The OpenSSH server is used for shell access and secure copy to the NSO Linux host for NSO version upgrade purposes. The NSO built-in SSH server provides CLI and NETCONF access to NSO.

The NSO services require Python.

To fulfill the tailf-hcc server dependencies, the iproute2 utilities and sudo packages are installed. See the section called “Dependencies” in the tailf-hcc package chapter for details on the dependencies.

The rsyslog package enables storing an NSO log file from several NSO logs locally and forwarding some logs to the manager.

The arp command from the net-tools and iputils (ping) packages has been added for demonstration purposes.

Create the ncsadmin and ncsoper Linux user groups.

Create and add the admin and oper Linux users to their respective groups.

Perform a system installation of NSO that runs NSO as the admin user.

The admin user is granted access to run the ip command from the vipctl script as root using the sudo command as required by the tailf-hcc package.

The cmdwrapper NSO program gets access to run the scripts executed by the generate-token action for generating RESTCONF authentication tokens as the current NSO user.

Password authentication is set up for the read-only oper user for use with NSO only, which is intended for WebUI access.

The root user is set up for Linux shell access only.

The NSO installer, tailf-hcc package, application YANG modules, scripts for generating and authenticating RESTCONF tokens, and scripts for running the demo are all available to the NSO and manager containers.

admin user permissions are set for the NSO directories and files created by the system install, as well as for the root, admin, and oper home directories.

The ncs.crypto_keys are generated and distributed to all nodes.

For HA Raft, TLS certificates are generated for all nodes.

The initial NSO configuration, ncs.conf, is updated and in sync (identical) on the nodes.

The SSH servers are configured to allow only SSH public key authentication (no password). The oper user can use password authentication with the WebUI but has read-only NSO access.

The oper user is denied access to the Linux shell.

The admin user can access the Linux shell and NSO CLI using public key authentication.

New keys for all users are distributed to the HA cluster nodes and the manager node when the HA cluster is initialized.

The OpenSSH server and the NSO built-in SSH server use the same private and public key pairs located under ~/.ssh/id_ed25519, while the manager public key is stored in the ~/.ssh/authorized_keys file for both NSO nodes.

Host keys are generated for all nodes to allow the NSO built-in SSH and OpenSSH servers to authenticate the server to the client.

Each HA cluster node has its own unique SSH host keys stored under ${NCS_CONFIG_DIR}/ssh_host_ed25519_key. The SSH client(s), here the manager, has the keys for all nodes in the cluster paired with the node's hostname and the VIP address in its /root/.ssh/known_hosts file.

The host keys, like those used for client authentication, are generated each time the HA cluster nodes are initialized. The host keys are distributed to the manager and nodes in the HA cluster before the NSO built-in SSH and OpenSSH servers are started on the nodes.

As NSO runs in containers, the environment variables are set to point to the system install directories in the Docker Compose .env file.

NSO run as the non-root admin user and, therefore, the ncs command is used to start NSO instead of the /etc/init.d/ncs and /etc/profile.d scripts. The environment variables are copied to a .pam_environment file so that the root and admin users can set the required environment variables when those users access the shell via SSH.

The start script is installed as part of the NSO system install, and it can be customized if you would like to use it to start NSO. The available NSO start script variants can be found under /opt/ncs/current/src/ncs/package-skeletons/etc. The scripts may provide what you need and can be used as a starting point.

If you are running NSO as the root user and using systemd, the init.d script can converted for use with systemd. Example:

$ mkdir -p /etc/init.d
$ ./nso-NSO_VERSION.linux.x86_64.installer.bin --system-install
$ cp /etc/init.d/ncs /etc/rc.d/init.d/
$ systemctl daemon-reload
$ echo "ExecReload=/etc/rc.d/init.d/ncs reload" << /run/systemd/generator.late/ncs.service
$ cp /run/systemd/generator.late/ncs.service /etc/systemd/system/
$ systemctl start ncs

The OpenSSH sshd and rsyslog daemons are started.

The packages from the package store are added to the ${NCS_RUN_DIR}/packages directory before finishing the initialization part in the root context.

The NSO smart licensing token is set.

The NSO IPC socket is configured in ncs.conf to only listen to localhost 127.0.0.1 connections, which is the default setting.

By default, the clients connecting to the NSO IPC socket are considered trusted, i.e., no authentication is required, and the use of 127.0.0.1 with the /ncs-config/ncs-ipc-address IP address in ncs.conf to prevent remote access. See the section called “Security Considerations” and ncs.conf(5) in Manual Pages for more details.

/ncs-config/aaa/pam is set to enable PAM to authenticate users as recommended. All remote access to NSO must now be done using the NSO host's privileges. See ncs.conf(5) in Manual Pages for details.

Depending on your Linux distribution, you may have to change the /ncs-config/aaa/pam/service setting. The default value is common-auth. Check the file /etc/pam.d/common-auth and make sure it fits your needs. See ncs.conf(5) in Manual Pages for details.

Alternatively, or as a complement to the PAM authentication, users can be stored in the NSO CDB database or authenticated externally. See the section called “Authentication” for details.

RESTCONF token authentication under /ncs-config/aaa/external-validation is enabled using a token_auth.sh script that was added earlier together with a generate_token.sh script. See ncs.conf(5) in Manual Pages for details.

The scripts allow users to generate a token for RESTCONF authentication through, for example, the NSO CLI and NETCONF interfaces that use SSH authentication or the Web interface.

The token provided to the user is added to a simple YANG list of tokens where the list key is the username.

The token list is stored in the NSO CDB operational data store and is only accessible from the node's local MAAPI and CDB APIs. See the HA Raft and rule-based HA upgrade-l2/manager-etc/yang/token.yang file in the examples.

The NSO web server HTTPS interface should be enabled under /ncs-config/webui, along with /ncs-config/webui/match-host-name = true and /ncs-config/webui/server-name set to the hostname of the node, following security best practice. See ncs.conf(5) in Manual Pages for details.

      $ openssl x509 -in /etc/ncs/ssl/cert/host.cert -text -noout
      Certificate:
      Data:
      Version: 1 (0x0)
      Serial Number: 2 (0x2)
      Signature Algorithm: sha256WithRSAEncryption
      Issuer: C=US, ST=California, O=Internet Widgits Pty Ltd, CN=John Smith
      Validity
      Not Before: Dec 18 11:17:50 2015 GMT
      Not After : Dec 15 11:17:50 2025 GMT
      Subject: C=US, ST=California, O=Internet Widgits Pty Ltd
      Subject Public Key Info:
      .......

Disable /ncs-config/webui/cgi unless needed.

The NSO SSH CLI login is enabled under /ncs-config/cli/ssh/enabled. See ncs.conf(5) in Manual Pages for details.

The NSO CLI style is set to C-style, and the CLI prompt is modified to include the hostname under /ncs-config/cli/prompt. See ncs.conf(5) in Manual Pages for details.

    <prompt1>\u@nso-\H> </prompt1>
    <prompt2>\u@nso-\H% </prompt2>

    <c-prompt1>\u@nso-\H# </c-prompt1>
    <c-prompt2>\u@nso-\H(\m)# </c-prompt2>

NSO HA Raft is enabled under /ncs-config/ha-raft, and the rule-based HA under /ncs-config/ha. See ncs.conf(5) in Manual Pages for details.

Depending on your provisioned applications, you may want to turn /ncs-config/rollback/enabled off. Rollbacks do not work well with nano service reactive FASTMAP applications or if maximum transaction performance is a goal. If your application performs classical NSO provisioning, the recommendation is to enable rollbacks. Otherwise not. See ncs.conf(5) in Manual Pages for details.

Adding the admin user to the ncsadmin group and the oper user to the limited ncsoper group will ensure that the two users get properly authorized with NSO.

Not adding the root user to any group matching the NACM groups results in zero access, as no NACM rule will match, and the default in the aaa_init.xml file is to deny all access.

All CDB data is replicated from the leader/primary to the follower/secondary nodes.

If the leader/primary fails, a follower/secondary takes over and starts to act as leader/primary. This is how HA Raft works and how the rule-based HA variant of this example is configured to handle failover automatically.

At failover, tailf-hcc sets up a virtual alias IP address on the leader/primary node only and uses gratuitous ARP packets to update all nodes in the network with the new mapping to the leader/primary node.

Device timeouts. NSO has connect, read, and write timeouts for traffic between NSO and the managed devices. The default value may not be sufficient if devices/nodes are slow to commit, while some are sometimes slow to deliver their full configuration. Adjust timeouts under /devices/global-settings accordingly.

Service code timeouts. Some service applications can sometimes be slow. Adjusting the /services/global-settings/service-callback-timeout configuration might be applicable depending on the applications. However, the best practice is to change the timeout per service from the service code using the Java ServiceContext.setTimeout function or the Python data_set_timeout function.

At startup, check that NSO has been started using the $NCS_DIR/bin/ncs_cmd -c "wait-start 2" command.

Use the ssh command to verify SSH access to the NSO host and NSO CLI.

Check disk-usage using, for example, the df utility.

For example, use curl or the Python requests library to verify that the RESTCONF API is accessible.

Check that the NETCONF API is accessible using, for example, the $NCS_DIR/bin/netconf-console tool with a hello message.

Verify the NSO version using, for example, the $NCS_DIR/bin/ncs --version or RESTCONF /restconf/data/tailf-ncs-monitoring:ncs-state/version.

Check if HA is enabled using, for example, RESTCONF /restconf/data/tailf-ncs-monitoring:ncs-state/ha.

Have all users that need access to NSO authenticated through Linux PAM. This may then be through /etc/passwd. Avoid storing users in CDB.

Given the default NACM authorization rules, you should have three different types of users on the system.