[[PageOutline]] == Tulane Box Accounts == In your Cypress session, you can transfer files between your [https://tulane.box.com Tulane Box account] and Cypress on the command line using the '''rclone''' command, which is available via the module '''rclone'''. (See Option 1 of 2 below and [[cypress/ModuleCommand|Module command]]). In order to use rclone on Cypress, you must first create the config file ''~/.config/rclone/rclone.conf'' on your Cypress account. You can generate your ''rclone.conf'' file either directly in your Cypress session or by creating it on your local machine and then copying it to your Cypress account via one of the following options. ==== Option 1 of 2: Generating Your rclone.conf File In Your Cypress Session ==== 1. Login to Cypress from your local machine with X11 forwarding enabled. (See [[cypress/SshUsage#TunnelingXWindowsSessionviaSSH|X11 Forwarding]].) You should confirm that X11 forwarding is enabled using the '''xterm''' command on Cypress as in the following. This will cause a new, separate terminal window to open on your local machine, which you can close by typing exit in that new terminal window. {{{ [tulaneID@cypress1 ~]$ xterm }}} 2. In your Cypress session enter the following commands. {{{ [tulaneID@cypress1 ~]$ module load rclone [tulaneID@cypress1 ~]$ rclone config }}} 3. At the prompt '''...No remotes found - make a new one...''', enter {{{n}}} for '''New remote'''. 4. At the prompt '''name>''', choose a meaningful name, e.g, {{{TU-Box}}}, to be used for future connection to your Tulane Box account from your Cypress command line session. 5. At the prompt '''Type of storage to configure...''', enter {{{6}}} for '''Box'''. 6. At the prompt '''...Box App Client Id...''', respond by pressing {{{Enter}}} for the default. 7. At the prompt '''...Box App Client Secret...''', respond by pressing {{{Enter}}} for the default. 8. At the prompt '''Edit advanced config? (y/n)''', enter {{{n}}} for '''No'''. 9. At the prompt '''Remote config - Use auto config?...''', enter {{{y}}} for '''Yes'''. 10. Observe the message ending in '''Waiting for code ...''', and wait for a separate browser window to open on your local machine with a Box web page. Depending on your network latency, you may experience a delay of 1 minute or more. In the Box web page, click on the link '''Use Single Sign On (SSO)'''. 11. In the next Box web page, enter your full Tulane email address and click on '''Authorize'''. 12. This will take you to the Tulane login web page, wherein you can complete your Tulane login information and click on '''Sign in'''. 13. This final web page should show the message '''Success! - All done. Please go back to rclone''', at which point you can close the browser window and return to your Cypress session command line window. 14. Back in the command line window, observe the new message '''Got code''' followed by your new token information ending with a prompt to which you can respond {{{y}}} for '''Yes this is OK'''. 15. The resulting response shows the list '''Current remotes:''', including your new Box remote, e.g., '''TU-Box''' followed by a prompt to which you can respond {{{q}}} for '''Quit config'''. 16. Observe the newly created config file '''~/.config/rclone/rclone.conf''' {{{ [tulaneID@cypress1 ~]$ ls -l ~/.config/rclone/ total 8 -rw------- 1 tulaneID group 233 Jun 29 20:00 rclone.conf [tulaneID@cypress1 ~]$ cat ~/.config/rclone/rclone.conf [TU-Box] type = box token = {"access_token":"ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567","token_type":"bearer","refresh_token":"ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890abcdefghijklmnopqrstuvwxyz12","expiry":"2020-06-29T21:03:28.009439309-05:00"} }}} at which point you're ready to run rclone on Cypress. (See '''Running rclone on Cypress''' below.) ==== Option 2 of 2: Generating Your rclone.conf File Using Your Local Machine ==== 1. Download, install, and configure rclone on your local machine by following the instructions for Windows, Mac, or Linux as appropriate starting from the [https://rclone.org/install/ rclone installation site.] a. During the above configuration, you will be prompted for a name for the Box remote system, e.g. '''TU-Box'''. b. Make note of your response for the name of the Box remote system for future reference to be used in your invocations of the rclone command. 2. Once you've configured rclone on your local machine, locate the resulting file '''rclone.conf''' on your local machine. a. On Mac and Linux, e.g., look for the file '''~/.config/rclone/rclone.conf''' b. On Windows 8 (or under) , e.g., look for the file '''%userprofile%\.config\rclone\rclone.conf''' 3. Create as needed the directory '''~/.config/rclone''' on your Cypress account. 4. Copy the resulting file '''rclone.conf''' from your local machine to your Cypress directory '''~/.config/rclone'''. See [[cypress/FileTransfer|File Transfer]]. 5. For security purposes, be sure to confirm that only you have ''exclusive'' read/write privileges to the resulting file '''~/.config/rclone/rclone.conf''' on Cypress. See the ''ls'' and ''chmod'' commands under [[cypress/BasicLinuxComands|Linux Commands]] Once you have created the config file '''~/.config/rclone/rclone.conf''' on Cypress, you can run rclone via the following. ==== Running rclone on Cypress ==== The following examples assume that you've already performed the configuration as above using the remote name '''TU-Box''' for your Tulane Box account. 1. If you haven't done so already from the above, load the module rclone. {{{ [tulaneID@cypress1 ~]$ module load rclone }}} 2. To list the entire contents in your Tulane Box account... {{{ [tulaneID@cypress1 ~]$ rclone ls TU-Box: }}} 3. To list the contents of a top-level directory in your Tulane Box account... {{{ [tulaneID@cypress1 ~]$ rclone ls TU-Box: }}} 4. To copy the contents of a top-level directory in your Tulane Box account to your current Cypress working directory... {{{ [tulaneID@cypress1 ~]$ rclone copy TU-Box: . }}} 5. To show progress using the '''-P''' option while copying the (~5GB) file bigFile.txt in your current Cypress working directory to a top-level directory in your Tulane Box account... {{{ [tulaneID@cypress1 ~]$ ls -l bigFile.txt -rw-r--r-- 1 tulaneID group 5000000000 Oct 25 2019 bigFile.txt [tulaneID@cypress1 ~]$ rclone -P copy bigFile.txt TU-Box: Transferred: 475.586M / 4.657 GBytes, 10%, 37.208 MBytes/s, ETA 1m55s Errors: 0 Checks: 0 / 0, - Transferred: 0 / 1, 0% Elapsed time: 12.7s Transferring: * bigFile.txt: 9% /4.657G, 35.761M/s, 2m0s }}} 6. To simply list the available subcommands under rclone... {{{ [tulaneID@cypress1 ~]$ rclone --help }}} 7. To show the help message - including a detailed explanation and helpful examples - for the rclone copy subcommand... {{{ [tulaneID@cypress1 ~]$ rclone copy --help }}} 8. To list the available flags used with the rclone command... (See also [[Box#BoxTransferRateLimits|Box Transfer Rate Limits]] below.) {{{ [tulaneID@cypress1 ~]$ rclone help flags }}} ==== Box Transfer Rate Limits ==== Depending on the number of files contained in the directory of your command {{{rclone copy }}}, many of those files may fail to transfer with error '''429 Too Many Requests'''. To protect quality of service, Box accounts enforce the following transfer rate limits, though with 2 additional retries for each failed transfer request by default. (See [https://developer.box.com/guides/api-calls/permissions-and-errors/rate-limits Box Rate Limits].) 1. 1000 API requests per minute, per user (for Box API requests other than file uploads) 2. 240 file upload requests per minute, per user (for file uploads to Box specifically) To avoid exceeding the above mentioned rate limits, you can limit your HTTP transactions per second by including the rclone flag '''--tpslimit''' in your rclone command as in the following. (See '''--tpslimit''' and other flags for rclone using the command {{{rclone help flags}}} for details.) 1. '''Downloading files:''' To limit your rate (to 1000 requests per minute) of downloading files from a top level directory of your Tulane Box account to your current Cypress working directory... {{{ [tulaneID@cypress1 ~]$ rclone copy TU-Box: . --tpslimit 16.6 # 1000 per minute or 16.6 per second }}} 1. '''Uploading files:''' To limit your rate (to 240 requests per minute) of uploading files to a top level directory of your Tulane Box account from your desired Cypress directory... {{{ [tulaneID@cypress1 ~]$ rclone copy TU-Box: --tpslimit 16.6 # 1000 per minute or 16.6 per second }}}