Tulane Box Accounts

In your Cypress session, you can transfer files between your ​Tulane Box account and Cypress on the command line using the rclone command, which is available via the module rclone. (See Option 1 of 2 below and Module command).

In order to use rclone on Cypress, you must first create the config file ~/.config/rclone/rclone.conf on your Cypress account. You can generate your rclone.conf file either directly in your Cypress session or by creating it on your local machine and then copying it to your Cypress account via one of the following options.

Option 1 of 2: Generating Your rclone.conf File In Your Cypress Session with X11 Forwarding Enabled

1. Login to Cypress from your local machine with X11 forwarding enabled. (See X11 Forwarding.) You should confirm that X11 forwarding is enabled using the xterm command on Cypress as in the following. This will cause a new, separate terminal window to open on your local machine, which you can close by typing exit in that new terminal window.

   [tulaneID@cypress1 ~]$ xterm
   

2. In your Cypress session enter the following commands.

   [tulaneID@cypress1 ~]$ module load rclone
   [tulaneID@cypress1 ~]$ rclone config
   

3. At the prompt …No remotes found - make a new one…, enter n for New remote.
4. At the prompt name>, choose a meaningful name, e.g, TU-Box, to be used for future connection to your Tulane Box account from your Cypress command line session.
5. At the prompt Type of storage to configure…, enter 6 for Box.
6. At the prompt …Box App Client Id…, respond by pressing Enter for the default.
7. At the prompt …Box App Client Secret…, respond by pressing Enter for the default.
8. At the prompt Edit advanced config? (y/n), enter n for No.
9. At the prompt Remote config - Use auto config?…, enter y for Yes.
10. Observe the message ending in Waiting for code …, and wait for a separate browser window to open on your local machine with a Box web page. Depending on your network latency, you may experience a delay of 1 minute or more. In the Box web page, click on the link Use Single Sign On (SSO).
11. In the next Box web page, enter your full Tulane email address and click on Authorize.
12. This will take you to the Tulane login web page, wherein you can complete your Tulane login information and click on Sign in.
13. This final web page should show the message Success! - All done. Please go back to rclone, at which point you can close the browser window and return to your Cypress session command line window.
14. Back in the command line window, observe the new message Got code followed by your new token information ending with a prompt to which you can respond y for Yes this is OK.
15. The resulting response shows the list Current remotes:, including your new Box remote, e.g., TU-Box followed by a prompt to which you can respond q for Quit config.
16. Observe the newly created config file ~/.config/rclone/rclone.conf

    [tulaneID@cypress1 ~]$ ls -l ~/.config/rclone/
    total 8
    -rw------- 1 tulaneID group 233 Jun 29 20:00 rclone.conf
    [tulaneID@cypress1 ~]$ cat ~/.config/rclone/rclone.conf
    [TU-Box]
    type = box
    token = {"access_token":"ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567","token_type":"bearer","refresh_token":"ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890abcdefghijklmnopqrstuvwxyz12","expiry":"2020-06-29T21:03:28.009439309-05:00"}
    
    

    at which point you're ready to run rclone on Cypress. (See Running rclone on Cypress below.)

Option 2 of 2: Generating Your rclone.conf File Using Your Local Machine

1. Download, install, and configure rclone on your local machine by following the instructions for Windows, Mac, or Linux as appropriate starting from the ​rclone installation site.
   1. During the above configuration, you will be prompted for a name for the Box remote system, e.g. TU-Box.
   2. Make note of your response for the name of the Box remote system for future reference to be used in your invocations of the rclone command.
2. Once you've configured rclone on your local machine, locate the resulting file rclone.conf on your local machine.
   1. On Mac and Linux, e.g., look for the file ~/.config/rclone/rclone.conf
   2. On Windows 8 (or under) , e.g., look for the file %userprofile%\.config\rclone\rclone.conf
3. Create as needed the directory ~/.config/rclone on your Cypress account.
4. Copy the resulting file rclone.conf from your local machine to your Cypress directory ~/.config/rclone. See File Transfer.
5. For security purposes, be sure to confirm that only you have exclusive read/write privileges to the resulting file ~/.config/rclone/rclone.conf on Cypress. See the ls and chmod commands under Linux Commands

Once you have created the config file ~/.config/rclone/rclone.conf on Cypress, you can run rclone via the following.

Running rclone on Cypress

The following examples assume that you've already performed the configuration as above using the remote name TU-Box for your Tulane Box account.

1. If you haven't done so already from the above, load the module rclone.

   [tulaneID@cypress1 ~]$ module load rclone
   

2. To list the entire contents in your Tulane Box account…

   [tulaneID@cypress1 ~]$ rclone ls TU-Box:
   

3. To list the contents of a top-level directory in your Tulane Box account…

   [tulaneID@cypress1 ~]$ rclone ls TU-Box:<top-level-directory>
   

4. To copy the contents of a top-level directory in your Tulane Box account to your current Cypress working directory…

   [tulaneID@cypress1 ~]$ rclone copy TU-Box:<top-level-directory> .
   

5. To show progress using the -P option while copying the (~5GB) file bigFile.txt in your current Cypress working directory to a top-level directory in your Tulane Box account…

   [tulaneID@cypress1 ~]$ ls -l bigFile.txt
   -rw-r--r-- 1 tulaneID group 5000000000 Oct 25  2019 bigFile.txt
   [tulaneID@cypress1 ~]$ rclone -P copy bigFile.txt TU-Box:<top-level-directory>
   Transferred:      475.586M / 4.657 GBytes, 10%, 37.208 MBytes/s, ETA 1m55s
   Errors:                 0
   Checks:                 0 / 0, -
   Transferred:            0 / 1, 0%
   Elapsed time:       12.7s
   Transferring:
    *                                   bigFile.txt:  9% /4.657G, 35.761M/s, 2m0s
   

6. To simply list the available subcommands under rclone…

   [tulaneID@cypress1 ~]$ rclone --help
   

7. To show the help message - including a detailed explanation and helpful examples - for the rclone copy subcommand…

   [tulaneID@cypress1 ~]$ rclone copy --help
   

8. To list the available flags used with the rclone command… (See also Box Transfer Rate Limits below.)

   [tulaneID@cypress1 ~]$ rclone help flags
   

Box Transfer Rate Limits

Depending on the number of files contained in the <source> directory of your command rclone copy <source> <destination>, many of those files may fail to transfer with error 429 Too Many Requests. To protect quality of service, Box accounts enforce the following transfer rate limits, though with 2 additional retries for each failed transfer request by default. (See ​Box Rate Limits.)

1. 1000 API requests per minute, per user (for Box API requests other than file uploads)
2. 240 file upload requests per minute, per user (for file uploads to Box specifically)

To avoid exceeding the above mentioned rate limits, you can limit your HTTP transactions per second by including the rclone flag —tpslimit in your rclone command as in the following. (See —tpslimit and other flags for rclone using the command rclone help flags for details.)

1. Downloading files: To limit your rate (to 1000 requests per minute) of downloading files from a top level directory of your Tulane Box account to your current Cypress working directory…

   [tulaneID@cypress1 ~]$ rclone copy TU-Box:<top-level-directory> . --tpslimit 16.6 # 1000 per minute or 16.6 per second
   

2. Uploading files: To limit your rate (to 240 requests per minute) of uploading files to a top level directory of your Tulane Box account from your desired Cypress directory…

   [tulaneID@cypress1 ~]$ rclone copy TU-Box:<top-level-directory> <Cypress-directory> --tpslimit 16.6 # 1000 per minute or 16.6 per second