Moving Data between Clusters
Due to the age of Roaring Thunder, it will be decommissioned in 2024. It is important to move any active research data from Roaring Thunder to Innovator. If you no longer need research data, RCi can archive it for you, just put in a work order LINKGOESHERE.
Using Command Line Utilities (rsync / cp)
There are multiple ways to achieve file transfers and I will cover rsync and cp here, which are Linux command line utilities in this section.
Log into Roaring Thunder using SSH as you normally would and navigate to /home or /scratch, depending on the location of the data to be moved, in this example, I will move data from /home on RT to my /home folder on Innovator.
[john.doe@rtlogin ~]$ pwd
/home/john.doe
[john.doe@rtlogin ~]$ ls |grep c_example
c_example
I have a folder called c_example that I want to copy to my Innovator /home directory. Using rsync I would use the format:
[john.doe@rtlogin ~]$ rsync -avp c_example john.doe@jacks.local@innovator.sdstate.edu:/home/jacks.local/john.doe
Using Windows Utilities (WinSCP)
Tools such as WinSCP and others can be used to transfer data using the Windows environment
Using Globus
Globus offers a fast, reliable method for transferring large files and folders between storage endpoints. This guide will help you use Globus Connect Personal to transfer data from "RT" to "Innovator."
1. Log In and Load Globus Module
Log in to Innovator using your jacks.local account. Load the Globus module with the command:
module load globus
2. Create a Personal Endpoint
Set up your personal endpoint with the command:
globusconnectpersonal -setup
This will open a GUI window for you to log in to your Globus account. Here, you can name and describe your personal endpoint (e.g., "My Innovator Personal Endpoint").
Note: Log in using your university credentials (e.g., john.doe).
3. Activate Your Personal Endpoint
Start your personal endpoint with:
globusconnectpersonal -start &
You'll see a confirmation like [1] 170467.
Verify the status of your endpoint to ensure it's connected:
[john.doe@jacks.local@cllogin002 ~]$ globusconnectpersonal -status
Globus Online: connected
Transfer Status: idle
4. Initiate File Transfer
Access the Globus web app via a browser at globus.com. Log in to begin.
- Open two panels using the icon at the top right corner of the screen.
- In one panel, select the personal endpoint you created (e.g., "My Innovator Personal Endpoint").
- In the other, search for "South Dakota State RT," choose the university's mapped collection on RT, and navigate to your account (e.g.,
/home/john.doe).
- You can now drag and drop files and folders for transfer.
