How do I transfer data from Google Drive to Panasas (Tier 1 storage)?

You can transfer data from Google drive to panasas via the command line using rclone. First, configure a remote connection to UMN's Google drive using rclone, a step that only needs to be done once. Then, using that remote connection, you can transfer files from Google Drive to panasas. 

 

The following instructions assume that you are logged into MSI via command line, and working within an interactive job. 

 

UMN Google Drive: Configure the remote connection

 

Create (or modify) an rclone config file. This process only needs to be done once. The resulting config file is stored in your home directory.

Execute the following commands from an interactive command line job:
   
  module load rclone/1.64.1

  rclone config

 

  • This launches the rclone config script. Provide the answers below to set up your remote connection (called a “remote”). Note that you should provide a name that makes sense to you for the second entry, the name of your remote, shown in bold below:
     

    • n (new remote connection)

    • name> your_umn_gdrive (provide it a "remote" name that makes sense to you)

    • storage> 15 (google drive)

    • client_id> (leave blank)

    • client_secret> (leave blank)

    • scope> 1 (Full access all files, excluding Application Data Folder)

    • root_folder_id> (leave blank)

    • service_account_file> (leave blank)

    • Edit advanced config> No

    • Remote config> No (headless machine)

    • Please go to the following link> (copy/paste link into browser, authorize google drive to use UMN account, copy the verification code provided by Google, paste back into terminal prompt)

    • Configure this as a team drive> No (not team drive)

    • y (Yes this is OK)

    • q (quit)

 

Copy files from Google Drive to Panasas

 

Execute the following commands from an interactive command line job:

Copy entire folder called "data" from google drive to a new dir called "data" in my current working dir on panasas.

rclone copy -v your_umn_gdrive:Work_MSI/Projects/data ./data

Copy entire contents of a folder called "data" on google drive to my current working dir on panasas.

rclone copy -v your_umn_gdrive:Work_MSI/Projects/data .

Copy a single file located at the full path "Work_MSI/Projects/file.txt" on google drive to my current working dir on panasas.

rclone copy -v your_umn_gdrive:Work_MSI/Projects/file.txt .
Category: 
Data Storage