Multiprotocol File Transfer Utilities

These utilities provide a common OS and protocol agnostic framework for client-side file-based operations such as:

• Transferring a single file to or from a remote server,

• Changing permissions, renaming, deleting and obtaining details of single remote files,

• Obtaining a remote directory listing,

• Performing existence and stability checking of a single remote file.

This framework strives to eliminate the need for multiple similar file transfer packages, all doing the same thing slightly differently, and to harmonize on a common extensible architecture that provides a consistent look and feel to users.

Linux client support is provided out of the box and additional file utility support may be added for other OS by installing available external plugin packages.

These operations are supported across a variety of protocols, to the extent allowed by the protocol and OS themselves (for example, it is not possible to request a remote directory listing via tftp from a Linux client). It may not be possible to request a chmod on platform XYZ because its ftp protocol client may not have this capability.

Files and directories are specified in the compact and expressive URL form [3], such as:

protocol://server.domain.com:port/path/to/file_or_directory

If the file or directory is not specified in URL form, then local form [2] is assumed (for example, /path/to/file is internally converted to file:/path/to/file).

Abstraction by OS

All file operations are, by default, done relative to the local execution host. The following example shows a sample operation on a Linux client:

---------------------------------------------------------------------------
# Contents of testbed YAML file tb.yaml

testbed:
  servers:
    server_alias:
      server: myserver.domain.com
      address: 1.1.1.1
      credentials:
        ftp:
          username: my_username
          password: my_password
---------------------------------------------------------------------------

# Example of Linux based file operations
#
from pyats.utils.fileutils import FileUtils
from pyats.topology import loader
tb = loader.load('tb.yaml')


from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
futils.copyfile(
    source = 'file:/device/local/path/to/file',
    destination = 'ftp://myserver.domain.com/remote/path/to/file')

In order for file operations to be done relative to a remote device (instead of the local execution host), an external package providing file utilities plugins for this OS must be pip-installed. The following example shows a sample operation on such a host:

---------------------------------------------------------------------------
# Contents of testbed YAML file tb.yaml
devices:
    my_xyz_device:
        os: xyz
        connections:
            protocol: telnet
            ip: 1.2.3.4
            port: 5678

testbed:
  servers:
    server_alias:
      server: myserver.domain.com
      address: 1.1.1.1
      credentials:
        ftp:
          username: my_username
          password: my_password
---------------------------------------------------------------------------

# Example of remote device based file operations
# (external plugin required)
#
from pyats.utils.fileutils import FileUtils
from pyats.topology import loader
tb = loader.load('tb.yaml')

futils = FileUtils.from_device(tb.devices['my_xyz_device'])
futils.copyfile(
    source = 'file:/device/local/path/to/file',
    destination = 'ftp://myserver.domain.com/remote/path/to/file')

Multi-Homed Server Handling

In the cases when a server has multiple IP addresses specified, the plugin does a one-shot determination of which IP address is reachable (slow path) and then uses this cached address from then on (fast path).

Here’s an example on a linux plugin:

---------------------------------------------------------------------------
# Contents of testbed YAML file tb.yaml

testbed:
  servers:
    server_alias:
      server: myserver.domain.com
      address:
          - 1.1.1.1
          - 2.2.2.2
      credentials:
        ftp:
          username: my_username
          password: my_password
---------------------------------------------------------------------------

# Example of Linux based file operations
#
from pyats.utils.fileutils import FileUtils
from pyats.topology import loader
tb = loader.load('tb.yaml')


# Note the use of server alias here.
#
from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
futils.copyfile(
    source = 'file:/device/local/path/to/file',
    destination = 'ftp://server_alias/remote/path/to/file')

# The linux plugin pings the first address and, if reachable, uses that
# address instead of the specified hostname (server_alias).
# Otherwise, the next address in the list is pinged.
# If no address is reachable, the hostname used defaults to
# the server key (myserver.domain.com) if provided.

Linux Plugin

This plugin provides file-based operations on a Linux client (including Mac), and is the default plugin returned by a call to FileUtils() when os is not specified.

The server portion of any of this plugin’s URL inputs may be specified as a name having a valid DNS lookup or as an explicit IP address but if there is no corresponding entry provided in the testbed servers block, users restrict themselves to using only the protocols such as scp and sftp that support local key-based authentication.

Examples

Here are some examples of file-based operations from a Linux client:

---------------------------------------------------------------------------
# Contents of testbed YAML file tb.yaml
testbed:
  servers:
    server_alias:
      server: myserver.domain.com
      address: 1.1.1.1
      credentials:
        default:
          username: my_username
          password: my_password

---------------------------------------------------------------------------
# Examples
# --------
#
from pyats.utils.fileutils import FileUtils
from pyats.topology import loader
tb = loader.load('tb.yaml')


# This with statement ensures that any sessions are automatically closed
# if something goes wrong.
with FileUtils(testbed=tb) as futils:
    # Copy local file to remote location (note the two ways of specifying server name):
    futils.copyfile(
        source = '/local/path/to/file',
        destination = 'ftp://server_alias/remote/path/to/file')

    futils.copyfile(
        source = 'file:///local/path/to/file',
        destination = 'tftp://myserver.domain.com/remote/path/to/file',
        timeout_seconds=80)


    # Copy remote file to local location, specifying the server via its address:
    futils.copyfile(
        source = 'scp://1.1.1.1/path/to/file',
        destination = '/local/path/to/file')

    # Copy remote file to a relative local location, when the local file
    # does not have a leading slash:
    # NOTE : Due to restrictions documented in RFC3986 section 3.3
    # it is not possible to specify the 'file://' prefix for this kind of operation.

    import os
    os.chdir('/local/path/to')
    futils.copyfile(
        source = 'scp://1.1.1.1/path/to/file',
        destination = 'file')

    # Copy remote file to a relative local location, when the local file
    # has a leading single or double dot.
    # NOTE : Due to restrictions documented in RFC3986 section 3.3
    # it is not possible to specify the 'file://' prefix for this kind of operation.

    import os
    os.chdir('/local/path/to')
    futils.copyfile(
        source = 'scp://1.1.1.1/path/to/file',
        destination = './file')

    # Copy remote file to a local location relative to the calling user's
    # home directory.
    # NOTE : Due to restrictions documented in RFC3986 section 3.3
    # it is not possible to specify the 'file://' prefix for this kind of operation.

    import os
    os.chdir('/local/path/to')
    futils.copyfile(
        source = 'scp://1.1.1.1/path/to/file',
        destination = '~/file')

    # Copy remote file to a local location relative to the named user's
    # home directory.
    # NOTE : Due to restrictions documented in RFC3986 section 3.3
    # it is not possible to specify the 'file://' prefix for this kind of operation.

    import os
    os.chdir('/local/path/to')
    futils.copyfile(
        source = 'scp://1.1.1.1/path/to/file',
        destination = '~nameduser/file')


    # Delete a remote file:
    futils.deletefile(target = 'sftp://myserver.domain.com/remote/path/to/file')


    # Change permissions of a remote file:
    import stat
    mode  = stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IXOTH
    futils.chmod(target = 'ftp://myserver.domain.com/remote/path/to/file', mode=mode)


    # Get details of a remote file (such as length and permissions):
    file_details = futils.stat(
        target = 'sftp://myserver.domain.com/remote/path/to/file')
    import stat
    stat.filemode(file_details.st_mode)
    '-rwxrwxrwx'


    # Obtain a remote directory listing (top-level only):
    futils.dir(target = 'ftp://myserver.domain.com/remote/path/to/')
    ['ftp://myserver.domain.com/remote/path/to/file',]


    # Rename a remote file:
    futils.renamefile(
        source = 'ftp://myserver.domain.com/remote/path/to/file',
        destination = 'ftp://myserver.domain.com/remote/path/to/renamed_file')


    # Check the existence of a remote file:
    futils.checkfile(target = 'ftp://myserver.domain.com/remote/path/to/file')


    # Check that a remote file exists and has a stable length (is not in the
    # middle of being written):

    futils.checkfile(
        target = 'ftp://myserver.domain.com/remote/path/to/file',
        check_stability=True)


    # Requesting an operation unsupported by the specified protocol
    # (expected to throw an exception):
    futils.checkfile(target = 'tftp://myserver.domain.com/remote/path/to/file')

Supported protocols

This section describes the supported protocols and the operations they offer.

An operation (such as copyfile) against a protocol (such as ftp) requires a protocol-named credential (such as ftp) to be declared at the server or testbed level. If not declared, then the default credential is used.

Please see Credential Password Modeling for details.

Protocol support by operation

Protocol support by operation

Operation	ftp	tftp [5]	scp [1]	sftp [1]
copyfile	Y	Y	Y	Y
deletefile	Y			Y
chmod	Y			Y
stat	Y			Y
dir	Y			Y
renamefile	Y			Y
checkfile	Y			Y
getspace			Y	Y

TFTP

Support for this protocol is only available when the execution server has the curl [4] system package installed. This package is commonly installed on most Linux distributions.

FTP

This protocol is supported natively.

Note

All operations on this plugin support an argument strip_leading_slash, which defaults to True, and thus causing the leading filename or path slash to be stripped before being sent to the remote server. Some servers require this to be set to False.

SCP

In order to enable support for this plugin, please execute the following command manually:

pip install scp paramiko

If local keys are available, they are also considered for authentication.

SFTP

In order to enable support for this plugin, please execute the following command manually:

pip install paramiko

If local keys are available, they are also considered for authentication.

API Guide

Common APIs

The following APIs are supported on any FileUtils instance:

from_device

Create a FileUtils instance from a device. This is typically used to access client-side file utilities on a non-Linux host, if such support has been made available via an external pip-installed plugin package.

The device is expected to have an os member, which is used to select the appropriate FileUtils plugin to be created.

futils = FileUtils.from_device(device=testbed.devices['my_xyz_device'])

Context Manager

A FileUtils instance may be used as a context manager, thus ensuring all sessions are properly closed whether or not the operation succeeded. Within the context block the session for a particular protocol is set up only once and is then reused thereafter.

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    file_details = futils.stat("sftp://server.domain.com/path/to/file")
    file2_details = futils.stat("sftp://server.domain.com/path/to/file2")

close

Close all sessions on a FileUtils instance.

Some protocol implementations (such as scp and sftp) keep their sessions open until they are explicitly closed.

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
try:
    file_details = futils.stat("sftp://server.domain.com/path/to/file")
    file2_details = futils.stat("sftp://server.domain.com/path/to/file2")
finally:
    futils.close()

checkfile

Check for remote file existence and (optionally) stability.

If check_stability is specified as True, multiple checks are done to ensure the file length is stable (it could be in the middle of being copied).

checkfile arguments

Argument	Type	Description
target	str	Remote file to check
check_stability	bool (default = False)	Whether or not to check file length stability.
max_tries	int (default = 3)	Maximum number of times to check the file before aborting.
delay_seconds	int (default = 2)	Delay between retries
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    futils.checkfile("sftp://server.domain.com/path/to/file")

Linux APIs

The following APIs are supported on any FileUtils Linux instance:

copyfile

Copy a single file either from local to remote or remote to local.

Remote to remote transfers are not supported (the user is expected to make multiple calls to do this).

Local to local transfers are not supported.

copyfile arguments

Argument	Type	Description
source	str	Source file for copy
destination	str	Destination file for copy
timeout_seconds	int (default = 1200)	Copy timeout [6]
quiet	bool (default = False)	quiet mode to suppress printing of copy progress

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    futils.copyfile(
        source = 'scp://1.1.1.1/path/to/file',
        destination = '/local/path/to/file')

deletefile

Delete a single remote file.

deletefile arguments

Argument	Type	Description
target	str	File to delete
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    futils.deletefile("sftp://server.domain.com/path/to/file")

dir

Retrieve filename URLs contained in the top level of a remote directory.

dir arguments

Argument	Type	Description
target	str	Directory whose contents are to be retrieved
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    futils.dir("sftp://server.domain.com/path/to/")

['sftp://server.domain.com/path/to/file',]

stat

Retrieve details of a remote file in a structure similar to that of os.stat.

stat arguments

Argument	Type	Description
target	str	File whose details are to be retrieved
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    file_details = futils.stat("sftp://server.domain.com/path/to/file")

    import stat
    print(stat.filemode(file_details.st_mode))

'-rwxrwxrwx'

chmod

Change the permissions of a remote file.

chmod arguments

Argument	Type	Description
target	str	File whose permissions are to be changed
mode	int	File permissions (same format as os.chmod)
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    import stat
    mode  = stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IXOTH
    futils.chmod(target = 'ftp://myserver.domain.com/remote/path/to/file', mode=mode)

renamefile

Rename a single remote file.

renamefile arguments

Argument	Type	Description
source	str	Remote file to rename
destination	str	New remote file name
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    futils.renamefile(
        source = 'ftp://myserver.domain.com/remote/path/to/file',
        destination = 'ftp://myserver.domain.com/remote/path/to/renamed_file')

getspace

Get the available disk space at target directory in bytes.

getspace arguments

Argument	Type	Description
target	str	Directory location to check available disk space
timeout_seconds	int (default = 60)	Connection timeout

from pyats.utils.fileutils import FileUtils
with FileUtils(testbed=tb) as futils:
    futils.getspace("sftp://server.domain.com/path/to/")

29832314880

Developer’s Guide

This section gives guidance on how to develop a FileUtils plugin for a device with a non-Linux OS.

File operations are expected to be performed relative to (ie. via running commands on) the device, Users are expected to first connect to the device and then instantiate the device’s FileUtils plugin using FileUtils.from_device.

Suggested Package File Layout

|-- .gitignore                        Tell git to ignore many common file types
|-- README.rst                        Points to src/myfutils/README.rst
|-- DESCRIPTION.rst                   Long description of package
|-- MANIFEST.in                       Extra files to be packaged
|-- setup.py                          For PyPi distribution of your package and
|                                     unit test execution.
|-- tests                             Sym-link to the src/myfutils/tests folder
|-- docs                              Sphinx online documentation for your package
|   |-- README.rst
|   `-- changelog
`-- src                               All sources for package myfutils
    |-- __init__.py                   Module declaration file
    `-- myfutils
        |-- __init__.py               Module declaration file
        |-- README.rst                Instructions to the installing user.
        |-- fileutils.py              Top-level fileutils, inherits from
        |                             ats.utils.fileutils.FileUtils
        |-- plugins                   All FileUtils OS plugins are under this directory
        |   |-- xyz                   Plugin module providing support for OS xyz
        |       |-- __init__.py       Module declaration file
        |       |-- fileutils.py      Fileutils module for OS xyz
        |       `-- ftp               Optional subplugin (child) module providing
        |           |                 support for protocol ftp on OS xyz
        |           `-- __init__.py   Module declaration file
        |           `-- fileutils.py  Fileutils module providing ftp services
        |                             for OS xyz
        |-- tests                     Unit tests for the package

Protocol-level Abstraction

A parent module advertises file utilities for a specific OS and implements user-facing operations defined by the base class.

If it has any contained packages they are treated as subplugins that a parent may instantiate via get_child. In this way, a parent may delegate processing of a file operation to a protocol-specific child object which is created only when the user invokes an operation against that protocol.

When a user executes from pyats.utils.fileutils import FileUtils all advertised OS-specific plugins (and any subplugins underneath them) are automatically loaded.

A child object is expected to both be contained by and inherit from the parent class. A child object has access to its parent object via its parent member (which is set to None on a parent object). A parent object has access to its child objects via its children member.

Every child must implement each user-facing operation implemented by its parent, and raise a NonImplementedError if it does not support the operation.

The first time a child is allocated for a given protocol by calling get_child it is instantiated and added to a cache. Subsequent calls by the parent to get_child return the cached object. This allows a child to maintain session state if required.

It is possible to remove a child from the cache so the next time get_child is called a new object is instantiated.

Package Designer’s Responsibilities

• Define one or more OS-specific plugins and ensure they inherit from pyats.utils.fileutils.FileUtils.

• Ensure plugins implementing the dir operation return a list of files expressed in URL form.

• Define a plugin point for each provided OS fileutils implementation to allow it to be automatically loaded by the core and be made accessible to users:

  # setup.py
  ...
  # console entry point
  entry_points = {
      'pyats.utils.fileutils.plugins' : [
          'xyz = myfutils.plugins.xyz',
      ],
  },
  

• Ensure each input URL is checked for remote/local and refuse any operations that are not supported.

• Parse each input URL with urllib.parse.urlparse and ignore unsupported parts with a warning (for example, embedded username/password, fragments or query parameters).

• Implement the close method and, if children have been allocated via get_child delegate the close to each child.

Sample Implementation

Here is a sample implementation of a FileUtils plugin for platform XYZ and protocol ftp.

The parent class:

from pyats.utils.fileutils import FileUtils as FileUtilsBase

class FileUtils(FileUtilsBsae):
  DEFAULT_COPY_TIMEOUT_SECONDS = 1200

  def close(self):
      """ Deallocate any resources being held.  """
      for child_name, child_obj in self.children.items():
          child_obj.close()


  def copyfile(self, source, destination,
          timeout_seconds = DEFAULT_COPY_TIMEOUT_SECONDS,
          *args, **kwargs):
      """ Copy a file to/from a remote server. """

      from_scheme = self.get_scheme(source)
      to_scheme = self.get_scheme(destination)

      from_scheme_is_local = self.is_local(source)
      to_scheme_is_local = self.is_local(destination)

      if from_scheme_is_local and to_scheme_is_local:
          raise Exception("fileutils module {} does not allow "
              "copying between two local files.".format(self.__module__))

      if not from_scheme_is_local and not to_scheme_is_local:
          raise Exception("fileutils module {} does not allow "
              "copying between two remote files.".format(self.__module__))


      abstraction_scheme = to_scheme if from_scheme_is_local else from_scheme

      # Get implementation
      child = self.get_child(abstraction_scheme, **kwargs)

      # Execute copy
      return child.copyfile(source, destination, timeout_seconds,
          *args, upload=from_scheme_is_local, **kwargs)

The child (protocol-specific) class:

from urllib.parse import urlparse
from .. import FileUtils as FileUtilsXyzBase

class FileUtils(FileUtilsXyzBase):

    def copyfile(self, source, destination,
            timeout_seconds = DEFAULT_COPY_TIMEOUT_SECONDS,
            *args, upload, **kwargs):

    from_parsed_url = urlparse(source)
    to_parsed_url = urlparse(destination)
    if upload:
        from_path = from_parsed_url.path

        to_server_name = to_parsed_url.hostname
        to_parsed_port = to_parsed_url.port
        to_path = to_parsed_url.path
        server_name = to_server_name
        port = to_parsed_port
    else:
        to_path = to_parsed_url.path
        from_server_name = from_parsed_url.hostname
        from_parsed_port = from_parsed_url.port
        from_path = from_parsed_url.path

        server_name = from_server_name
        port = from_parsed_port

    # Get auth details
    username, password = self.get_auth(server_name)

    # Transfer the file by executing commands on the device.
    if upload:
        upload_ftp_file(
            from_path=from_path,
            to_path=to_path,
            device=self.parent.device,
            username=username,
            password=password,
            timeout=timeout_seconds)
    else:
        download_ftp_file(
            from_path=from_path,
            to_path=to_path,
            device=self.parent.device,
            username=username,
            password=password,
            timeout=timeout_seconds)

API Guide for Plugin Developers

is_local

Returns True if the URL refers to a local resource.

is_local arguments

Argument	Type	Description
url	str	URL to check

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
assert futils.is_local("file:///path/to/local/file"

is_remote

Returns True if the URL refers to a remote (ie. non-local) resource.

is_remote arguments

Argument	Type	Description
url	str	URL to check

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
assert futils.is_remote("ftp://server.com/path/to/remote/file"

get_protocol

Returns the URL protocol (scheme).

get_protocol arguments

Argument	Type	Description
url	str	URL to parse

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
assert futils.get_protocol("ftp://server.com/path/to/remote/file") == 'ftp'

get_auth

Get authentication details from self.testbed

Returns (username, password) as strings or None if not found.

get_auth arguments

Argument	Type	Description
server_name_or_ip	str	Server name or alias or IP address.

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
username, password = futils.get_auth(server_name)

get_hostname

Get hostname details from self.testbed.

The hostname is set to:

• the contents of the address key if present in the testbed’s server block.

  • if multiple address is present (eg, the server block contains a list of addresses), the first network-reachable address (eg, responds to ping), is returned

• Otherwise, the contents of the server key if present in the testbed’s server block.

• Otherwise, the server alias.

• Otherwise, if the server block is not found in the testbed, None.

get_hostname arguments

Argument	Type	Description
server_name_or_ip	str	Server name or alias or IP address.

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
hostname = futils.get_hostname(server_name)

get_server_block

Get server block from self.testbed

Returns the server block dictionary.

get_server_block arguments

Argument	Type	Description
server_name_or_ip	str	Server name or alias or IP address.

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
server_block = futils.get_server_block(server_name)

get_child

Get a child FileUtils object under the current OS. If a child does not exist in the cache it is instantiated, otherwise it is returned from the cache.

To be called by a parent (OS-specific) FileUtils object.

The parent member of the child points back to the containing parent. The children member of the parent holds a list of instantiated children.

get_child arguments

Argument	Type	Description
abstraction_key	str	The name of the protocol being abstracted

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
futils_ftp = futils.get_child('ftp')
assert futils_ftp.parent is futils

remove_child

Remove a child FileUtils object from the cache.

To be called by a parent (OS-specific) FileUtils object.

All resources are first deallocated by calling the child’s close method. The child is then removed from the cache and deleted.

remove_child arguments

Argument	Type	Description
abstraction_key	str	The name of the protocol being abstracted

from pyats.utils.fileutils import FileUtils
futils = FileUtils(testbed=tb)
futils_ftp = futils.get_child('ftp')
assert futils_ftp.parent is futils

[1] (1,2)

Requires additional python dependencies to be manually pip-installed.

[2]

Please see Appendix B of RFC8089 for examples of how local files are expressed in URL form.

[3]

Please see RFC3986 for a standards-based discussion of URLs.

[4]

curl is a popular file transfer system package.

[5]

Requires additional system packages to be manually installed.

[6]

The connection timeout is calculated as a percentage of the copy timeout in order to keep the API simple.