INSTALL

Shift Installation and Configuration
====================================

1. Deployments

    Shift consists of four executable components:
    
        o shiftc    - the Shift client, which is invoked by users and other
                      client instances and must exist on hosts that initiate
                      transfers and hosts on which a transfer is to be
                      parallelized (henceforth called "client hosts")

        o shift-mgr - the Shift manager, which is invoked by client instances
                      and must exist either on all client hosts or on a host
                      accessible via ssh hostbased or publickey authentication
                      (henceforth called "manager hosts")

        o shift-aux - the Shift auxiliary utility, which is invoked by client
                      instances and is optional for basic functionality, but
                      must be installed on any hosts accessed as the source or
                      destination of a transfer initiated from a different host
                      (henceforth called "remote hosts") for efficient remote
                      file initialization, higher speed transfers via the
                      built-in fish/fish-tcp protocols, remote acl/xattr
                      preservation, remote lustre striping, and remote
                      verification (if msum from mutil is not installed)

        o shift-bin - the Shift binary helper, which is an optional component
                      invoked by shiftc and shift-aux to perform file
                      operations that cannot be implemented directly in pure
                      perl and/or cannot be executed efficiently in
                      batches by standard command line utilities.

    The main consideration for a Shift deployment is deciding which hosts will
    be the manager hosts.  In the simplest case, all client hosts are manager
    hosts.  If there is more than one client host, then the manager must be
    configured to store its metadata on a shared file system that supports
    locking.

    In a more complex multi-user, multi-host environment without a shared file
    system across all potential client hosts, one or more hosts must be
    designated as manager hosts.  Since the manager is not an active server
    and is simply a passive executable invoked by clients, the manager can be
    located on any host with ssh access.  These hosts must support either
    hostbased or publickey authentication, however, since Shift is an automated
    framework that performs actions without the user present.  The manager has
    a synchronization mechanism allowing multiple hosts without a shared file
    system to be used for redundancy.


2. Prerequisites

    2.1. Required

        o perl >= 5.8.5 (>= 5.10.1 is required for multi-threading support)
        o ssh (and sftp) access to manager/remote hosts via hostbased or
              publickey authentication

    2.2. Optional

        o Data::MessagePack (compiled version) - increase metadata performance
            (*highly recommended* for very large multi-host transfers)
            (https://metacpan.org/pod/Data::MessagePack)
        o IO::Socket::SSL - allows fish-tcp encryption with --secure
            (https://metacpan.org/pod/IO::Socket::SSL)
        o bbftp - high speed remote copy
            (http://doc.in2p3.fr/bbftp)
        o gnuplot >= 5.0 - allow display of --plot output
            (http://gnuplot.info)
        o mcp/msum >= 1.76.7 - high speed local copy/sum
            (http://mutil.sf.net)
        o mesh - lightweight single sign-on via ssh publickey authentication
            (http://mesh.sf.net)
        o rsync - bandwidth-efficient local/remote copy
            (http://rsync.samba.org)

    2.3. Invoked (standard on most systems or when specific file systems in use)

        o beegfs-ctl - get beegfs server information
        o chown - change symlink ownership
        o crontab - install cron jobs to recover from host/process failures
        o df - get file system utilization
        o dm{get,put} - recall and migrate DMF-managed files
        o fallocate - preallocate files
        o {get,set}facl - get and set file ACLs
        o {get,set}fattr - get and set file extended attributes
        o lfs - get and set Lustre striping
        o lspci - find 10GE adapters
        o mount - get file system information
        o ping - determine network latency
        o ps - find clients and PBS processes, and determine client CPU load
        o su - become non-root process to access manager during root transfers
        o sysctl - determine number of cpus on BSD
        o touch - change symlink modification time


3. Build (optional - linux only)

    An optional binary component called "shift-bin" can be built on
    linux systems to significantly improve the efficiency of some file
    operations.  These operations include equivalents to those provided
    by posix_fadvise, fallocate, {get,set}facl, {get,set}fattr, and lfs.
    Development headers for acls and xattrs are required on the system.
    This utility must be built differently depending on whether the
    system mounts lustre (and has access to the corresponding liblustreapi
    library) or not.

    3.1. Systems mounting lustre file systems

        cd c
        make lustre

    3.2. Systems not mounting lustre file systems

        cd c
        make nolustre


4. Installation

    Note that the following shows the exact files needed on each type
    of host.  Since the number of files is small, however, there is
    minimal penalty to simply installing them all on every host.  The
    installation commands assume that the optional shift-bin component
    has been built as needed for client and remote hosts.  

    4.1. Single-user installation

        Note that the user's home directory is used as the default install
        prefix in all examples, but can be changed to any other desired location
        as long as the corresponding bin directory is in the user's path.

        4.1.1. Client hosts

            install -m 700 perl/shiftc ~/bin/
            install -m 700 c/shift-bin ~/bin/
            install -m 600 doc/shiftc.1 ~/man/man1/

        4.1.2. Manager hosts

            install -m 700 perl/shift-mgr ~/bin/
            install -m 600 etc/shiftrc ~/.shiftrc

        4.1.3. Remote hosts (optional but recommended when possible)

            install -m 700 perl/shift-aux ~/bin/
            install -m 700 c/shift-bin ~/bin/

    4.2. Multi-user installation

        Note that /usr/local is used as the default install prefix in all
        examples, but can be changed to any other desired location as long
        as the corresponding bin directory is in the default system path.

        4.2.1. Client hosts

            install -m 755 perl/shiftc /usr/local/bin/
            install -m 755 c/shift-bin /usr/local/bin/
            install -m 644 doc/shiftc.1 /usr/local/man/man1/

        4.2.2. Manager hosts

            install -m 755 perl/shift-mgr /usr/local/bin/
            install -m 644 etc/shiftrc /etc/

        4.2.3. Remote hosts (optional but recommended when possible)

            install -m 755 perl/shift-aux /usr/local/bin/
            install -m 755 c/shift-bin /usr/local/bin/


5. Configuration

    5.1. Client hosts

        5.1.1. ~/.ssh/id_rsa (or similar)

            If hostbased authentication is not supported by client hosts,
            manager hosts, and/or remote hosts, publickey authentication must
            be used.  Clients must have access to private keys to access
            these systems, which may either be referenced explicitly via the
            --mgr-identity and --identity options for manager and remote
            hosts, respectively, or be added to an ssh agent on the client(s).
            Note that the private keys used for these options must not be
            protected by a passphrase.  The use of an ssh agent to access
            manager or remote hosts may be preferable from a security standpoint
            but comes with a drop in reliability as any failure of the agent or
            agent host leaves any associated transfers with no way to recover.

        5.1.2. ~/.ssh/authorized_keys

            If hostbased authentication is not supported to other client hosts
            for parallelization, publickey authentication must be used.  In this
            case, the public key(s) corresponding to the private key(s) used by
            clients (those named ~/.ssh/id* loaded into an ssh agent) must be
            added to the invoking user's authorized_keys file on other client
            hosts.

        5.1.3. ~/bin/shiftc (single-user) or /usr/local/bin/shiftc (multi-user)

            If the manager hosts differ from the client hosts, the manager
            host(s) can be hardcoded within the shiftc program in the
            "site-specific options" section.  This allows the client to be used
            without specifying the --mgr option every time.  For example, the
            line:

                $opts{"mgr"} = "mgr.example.com";

            would be equivalent to specifying the option "--mgr=mgr.example.com".
            In general, any shiftc command-line option --X can be hardcoded using:

                $opts{"X"} = "hardcoded_value";

            Those familiar with perl can add more complex logic (e.g. choosing
            which manager host out of a set of hosts will be used for each
            invocation).

    5.2. Manager hosts

        5.2.1. ~/.shiftrc (single-user) or /etc/shiftrc (multi-user)

            All items in the default config file should be reviewed.
            The only required setting is:

                user_dir

            If there are multiple manager hosts, the configured directory must
            either be a file system shared across all manager hosts that
            supports file locking or the setting:

                sync_host

            must be configured to sync the transfer metadata across two
            manager hosts.

            The transport options should definitely be reviewed to
            enable any higher performance transports that may be
            available.  Higher performance remote transports based on
            TCP connections (fish-tcp) and SSH connections (fish) are
            built-in, but require that the shift-aux executable exist
            on remote hosts.  Note that transports can be enabled on a
            case-by-case basic using the client --local and --remote
            options.

            To allow Shift to make parallelization decisions and enable
            functionality specific to certain remote file systems (e.g. Lustre
            striping), it is desirable to configure:

                mount_db

            with a database of host and file system information from within
            the local environment.  A template for producing this database is
            provided in the file "etc/shift-mounts.pl".  After the items
            indicated in the script are configured, it can be run
            once/periodically to initialize/update the file system database.

            To optimize access to remote hosts on the LAN, the setting:

                select_hook

            may be configured to specify how remote hosts should be selected
            when more than one is available.  This may include deciding which
            hosts are up, which are least loaded, which have the best
            connectivity to the given client host, etc.  A skeleton for a
            suitable selection hook is provided in the file
            "etc/shift-select.hook", but must be fleshed out based on
            site-specific knowledge and/or calls to the local load balancing
            infrastructure.  If not configured, hosts will be chosen randomly
            after a successful sshd ping test.

        5.2.2. ~/.ssh/authorized_keys

            If hostbased authentication is not supported to manager hosts,
            publickey authentication must be used.  In this case, the public
            key(s) corresponding to the private key(s) used by clients (either
            loaded into an ssh agent or specified via --mgr-identity) must be
            added to the appropriate user's authorized_keys file on manager
            hosts.  This user will either be the invoking user or the one
            specified by --mgr-user.


    5.3. Remote hosts

        5.3.1. ~/.ssh/authorized_keys

            If hostbased authentication is not supported to remote hosts,
            publickey authentication must be used.  In this case, the public
            key(s) corresponding to the private key(s) used by clients (either
            loaded into an ssh agent or specified via --identity) must be added
            to the appropriate user's authorized_keys file on remote hosts.
            This user will either be the invoking user or the one specified by
            --user.


6. Usage

    6.1. shiftc

        Client usage is detailed in the man page "doc/shiftc.1", which can be
        viewed with:
        
            nroff -man doc/shiftc.1 |less -r
            
        if not already in a manpath directory.  Basic usage is drop-in
        compatible with cp/scp with special consideration for the authentication
        options --mgr, --mgr-identity, --mgr-user, --identity, and --user.

    6.2. shift-mgr

        Normal users need not invoke shift-mgr directly as all relevant manager
        functionality is accessed indirectly through the client.  Additional
        capabilities are available for administrators, however, to aid in
        collecting usage stats and debugging information.  Below is a sampling
        of some of the most useful administrative shift-mgr commands.

        To see the status of a particular user X's transfers, run:

            shift-mgr --user=X --status

        To see the history of a particular user X's transfers, run:

            shift-mgr --user=X --history

        To see the status of running transfers across all users, run:

            shift-mgr --status --state=run

        To see detailed stats across the transfers of all users including a
        sampling of error messages, run:

            shift-mgr --stats

        This can be added to a cron job (normally at the interval of the
        "data_expire" setting in shiftrc) with a redirection to a file to
        periodically save usage stats.

        To see the metadata associated with a given user X's transfer T, run:

            shift-mgr --user=X --id=T --meta

        The metadata is log structured, so it is also possible to view the
        metadata at any number of steps S back in time using:

            shift-mgr --user=X --id=T --meta=S